[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:08 dist-test-slave-8vqf krb5kdc[30792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:08 dist-test-slave-8vqf krb5kdc[30792](info): set up 2 sockets
Sep 27 07:06:08 dist-test-slave-8vqf krb5kdc[30792](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 07:06:09.093686 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:09 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:09.270900 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42299
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:09.598673 30805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:09.599011 30805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:09.599102 30805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:09.606516 30805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:09.606698 30805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:09.606752 30805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:09.606810 30805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:09.606854 30805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:09.616956 30805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:42299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:09.618275 30805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:09.619681 30805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:09.632375 30811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.632470 30810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:09.632488 30813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:09.633713 30805 server_base.cc:1034] running on GCE node
I20260927 07:06:09.634617 30805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:09.636112 30805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:09.637365 30805 hybrid_clock.cc:648] HybridClock initialized: now 1790492769637324 us; error 53 us; skew 500 ppm
I20260927 07:06:09.637737 30805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:09 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:09.648522 30805 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:09.650204 30805 webserver.cc:460] Webserver started at http://127.30.16.126:42255/ using document root <none> and password file <none>
I20260927 07:06:09.651012 30805 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:09.651117 30805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:09.651386 30805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:09.654368 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "1afa771cad3842c2ba636dab58a44a25"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-8vqf"
server_key: "1b3312caf87a23bd03ea981a58073076"
server_key_iv: "1a31be50a486c380b5f1525250d4db02"
server_key_version: "encryptionkey@0"
I20260927 07:06:09.655170 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "1afa771cad3842c2ba636dab58a44a25"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-8vqf"
server_key: "1b3312caf87a23bd03ea981a58073076"
server_key_iv: "1a31be50a486c380b5f1525250d4db02"
server_key_version: "encryptionkey@0"
I20260927 07:06:09.660636 30805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 07:06:09.664287 30820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:09.665905 30805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:06:09.666109 30805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "1afa771cad3842c2ba636dab58a44a25"
format_stamp: "Formatted at 2026-09-27 07:06:09 on dist-test-slave-8vqf"
server_key: "1b3312caf87a23bd03ea981a58073076"
server_key_iv: "1a31be50a486c380b5f1525250d4db02"
server_key_version: "encryptionkey@0"
I20260927 07:06:09.666332 30805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:09.695533 30805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:09.700268 30805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:09.700591 30805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:09.719373 30805 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:42299
I20260927 07:06:09.719395 30871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:42299 every 8 connection(s)
I20260927 07:06:09.720897 30805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:09.724815 30872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:09.728857 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 30805
I20260927 07:06:09.729177 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:09.731242 30785 external_mini_cluster.cc:1428] Setting key 311938e0d250099729c0b230722d1a5c
I20260927 07:06:09.736300 30872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25: Bootstrap starting.
I20260927 07:06:09.739876 30872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:09.741036 30872 log.cc:824] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:09.744026 30872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25: No bootstrap required, opened a new log
I20260927 07:06:09.753015 30872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } }
I20260927 07:06:09.753372 30872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:09.753521 30872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afa771cad3842c2ba636dab58a44a25, State: Initialized, Role: FOLLOWER
I20260927 07:06:09.754251 30872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } }
I20260927 07:06:09.754540 30872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:09.754753 30872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:09.755002 30872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:09.757202 30872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } }
I20260927 07:06:09.757735 30872 leader_election.cc:302] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1afa771cad3842c2ba636dab58a44a25; no voters:
I20260927 07:06:09.758409 30872 leader_election.cc:288] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:09.758540 30877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 1 FOLLOWER]: Leader election won for term 1
Sep 27 07:06:09 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:09.760002 30877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [term 1 LEADER]: Becoming Leader. State: Replica: 1afa771cad3842c2ba636dab58a44a25, State: Running, Role: LEADER
I20260927 07:06:09.760612 30877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } }
I20260927 07:06:09.761039 30872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:09.763989 30879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1afa771cad3842c2ba636dab58a44a25. Latest consensus state: current_term: 1 leader_uuid: "1afa771cad3842c2ba636dab58a44a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } } }
I20260927 07:06:09.764478 30879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:09.765413 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1afa771cad3842c2ba636dab58a44a25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1afa771cad3842c2ba636dab58a44a25" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42299 } } }
I20260927 07:06:09.765707 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:09.768352 30882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:09.773066 30874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:09.735421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.30.16.126:42299)
0927 07:06:09.736263 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:09.736287 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:09.737547 (+ 1260us) server_negotiation.cc:407] Connection header received
0927 07:06:09.746687 (+ 9140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:09.746711 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:09.747146 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:09.747585 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:09.750623 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.751997 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.753831 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:09.754606 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:09.754832 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:09.761245 (+ 6413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:09.761255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:09.761270 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:09.761311 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:09.765810 (+ 4499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.767017 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.767026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.767034 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.767292 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:09.767892 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:09.767904 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:09.767907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:09.768742 (+ 835us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:09.769046 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:09.770766 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0927 07:06:09.771361 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":383,"threads_started":1}
I20260927 07:06:09.776122 30882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:09.785069 30882 catalog_manager.cc:1347] Generated new cluster ID: 866e9bea6fe44098bc06e1916f43765e
I20260927 07:06:09.785194 30882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:09.794137 30882 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:09.795710 30882 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:09.816772 30882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1afa771cad3842c2ba636dab58a44a25: Generated new TSK 0
I20260927 07:06:09.817862 30882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:09.867336 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:10.202770 30898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:10.203083 30898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:10.203254 30898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:10.210819 30898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:10.210943 30898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:10.211144 30898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:10.220985 30898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:10.222427 30898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:10.223850 30898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:10.237489 30903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:10.237576 30904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:10.237553 30906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:10.238438 30898 server_base.cc:1034] running on GCE node
I20260927 07:06:10.239118 30898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:10.240106 30898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:10.241309 30898 hybrid_clock.cc:648] HybridClock initialized: now 1790492770241259 us; error 37 us; skew 500 ppm
I20260927 07:06:10.241650 30898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:10 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492770, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:10.244879 30898 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:10.246335 30898 webserver.cc:460] Webserver started at http://127.30.16.65:37485/ using document root <none> and password file <none>
I20260927 07:06:10.247139 30898 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:10.247291 30898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:10.247618 30898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:10.250782 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "83a5f4f184a74dd8a4dd32f25de05446"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "4ed00c334ce403c4f1d5b98e2ace257a"
server_key_iv: "9e6248aab723663a65142893aab3715f"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.251596 30898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "83a5f4f184a74dd8a4dd32f25de05446"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "4ed00c334ce403c4f1d5b98e2ace257a"
server_key_iv: "9e6248aab723663a65142893aab3715f"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.256904 30898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:10.260542 30913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.262259 30898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:06:10.262490 30898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "83a5f4f184a74dd8a4dd32f25de05446"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "4ed00c334ce403c4f1d5b98e2ace257a"
server_key_iv: "9e6248aab723663a65142893aab3715f"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.262759 30898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:10.283728 30898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:10.288695 30898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:10.288967 30898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:10.290019 30898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:10.291584 30898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:10.291682 30898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.291776 30898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:10.291829 30898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.328055 30898 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41359
I20260927 07:06:10.328091 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41359 every 8 connection(s)
I20260927 07:06:10.329511 30898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:10.336174 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 30898
I20260927 07:06:10.336421 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:10.337195 30785 external_mini_cluster.cc:1428] Setting key 64fa261966ce29eedbff93a400e40f50
Sep 27 07:06:10 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492770, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:10.352820 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:10.332368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:35397 (local address 127.30.16.126:42299)
0927 07:06:10.332836 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:10.332841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:10.333787 (+ 946us) server_negotiation.cc:407] Connection header received
0927 07:06:10.335844 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:10.335849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:10.335925 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:10.336020 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:10.338202 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.339329 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.341111 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.341550 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.341705 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:10.346174 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:10.346179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:10.346183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:10.346218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:10.349125 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:10.350177 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:10.350181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:10.350184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:10.350321 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:10.350906 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:10.350910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:10.350912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:10.351195 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:10.351355 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:10.352352 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 07:06:10.352548 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":258,"threads_started":1}
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:10.354656 31029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:10.332904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42299 (local address 127.30.16.65:35397)
0927 07:06:10.333619 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:10.333657 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:10.335510 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:10.336331 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:10.336343 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:10.337119 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:10.337979 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:10.338000 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.339500 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.339505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:10.340944 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:10.340956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.341811 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.341820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:10.342092 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:10.342905 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:10.342947 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:10.345906 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:10.349290 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:10.349303 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:10.349327 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:10.350029 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:10.350456 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:10.350465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:10.350471 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:10.350771 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:10.351343 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:10.351355 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:10.351864 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 07:06:10.352219 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 07:06:10.352854 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":273,"threads_started":1}
I20260927 07:06:10.356313 31026 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42299
I20260927 07:06:10.356647 31026 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:06:10.357652 31026 heartbeater.cc:511] Master 127.30.16.126:42299 requested a full tablet report, sending...
I20260927 07:06:10.360147 30837 ts_manager.cc:194] Registered new tserver with Master: 83a5f4f184a74dd8a4dd32f25de05446 (127.30.16.65:41359)
I20260927 07:06:10.361855 30837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:35397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:10.380538 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:10.711892 31032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:10.712239 31032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:10.712389 31032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:10.719993 31032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:10.720155 31032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:10.720369 31032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:10.730417 31032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:10.731802 31032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:10.733233 31032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:10.747020 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:10.747136 31040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:10.747574 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:10.747973 31032 server_base.cc:1034] running on GCE node
I20260927 07:06:10.748648 31032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:10.749711 31032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:10.750946 31032 hybrid_clock.cc:648] HybridClock initialized: now 1790492770750906 us; error 35 us; skew 500 ppm
I20260927 07:06:10.751343 31032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:10 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492770, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:10.754881 31032 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:10.756421 31032 webserver.cc:460] Webserver started at http://127.30.16.66:43953/ using document root <none> and password file <none>
I20260927 07:06:10.757153 31032 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:10.757298 31032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:10.757619 31032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:10.760778 31032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "0fe0b416922e45e3b66954c1b4234579"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "f4903960383629e0b1d6ca2a9576df7f"
server_key_iv: "3f8b714a47d65a8065848d80ac005b1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.761642 31032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "0fe0b416922e45e3b66954c1b4234579"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "f4903960383629e0b1d6ca2a9576df7f"
server_key_iv: "3f8b714a47d65a8065848d80ac005b1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.767172 31032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:10.771133 31047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.772676 31032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:10.772854 31032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "0fe0b416922e45e3b66954c1b4234579"
format_stamp: "Formatted at 2026-09-27 07:06:10 on dist-test-slave-8vqf"
server_key: "f4903960383629e0b1d6ca2a9576df7f"
server_key_iv: "3f8b714a47d65a8065848d80ac005b1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:10.773066 31032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:10.811425 31032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:10.816025 31032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:10.816344 31032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:10.817394 31032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:10.818892 31032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:10.819002 31032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.819157 31032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:10.819227 31032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:10.853032 31032 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:46509
I20260927 07:06:10.853099 31159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:46509 every 8 connection(s)
I20260927 07:06:10.854408 31032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:10.858587 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31032
I20260927 07:06:10.858793 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:10.859333 30785 external_mini_cluster.cc:1428] Setting key deba134a121c03ca9bfce000bf5cf555
Sep 27 07:06:10 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492770, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:10.877393 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:10.857398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:34529 (local address 127.30.16.126:42299)
0927 07:06:10.857863 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:10.857870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:10.859137 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 07:06:10.861549 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:10.861554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:10.861618 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:10.861712 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:10.863970 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.864673 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.866803 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.867237 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.867407 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:10.871209 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:10.871214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:10.871217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:10.871249 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:10.874023 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:10.874826 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:10.874836 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:10.874838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:10.874961 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:10.875460 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:10.875469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:10.875472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:10.875779 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:10.875938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:10.876756 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 07:06:10.877006 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":215,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:10.878703 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:10.858018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42299 (local address 127.30.16.66:34529)
0927 07:06:10.858954 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:10.859010 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:10.861197 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:10.862018 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:10.862033 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:10.862880 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:10.863765 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:10.863795 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.864819 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.864827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:10.866631 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:10.866642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:10.867462 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:10.867469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:10.867854 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:10.868675 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:10.868697 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:10.870956 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:10.874166 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:10.874175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:10.874194 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:10.874699 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:10.875075 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:10.875081 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:10.875085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:10.875339 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:10.875915 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:10.875925 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:10.876375 (+ 450us) client_negotiation.cc:769] Sending connection context
0927 07:06:10.876651 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 07:06:10.877221 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":325,"threads_started":1}
I20260927 07:06:10.880306 31160 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42299
I20260927 07:06:10.880637 31160 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:10.881398 31160 heartbeater.cc:511] Master 127.30.16.126:42299 requested a full tablet report, sending...
I20260927 07:06:10.882916 30837 ts_manager.cc:194] Registered new tserver with Master: 0fe0b416922e45e3b66954c1b4234579 (127.30.16.66:46509)
I20260927 07:06:10.883740 30837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:34529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:10.902752 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:11.237036 31166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:11.237361 31166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:11.237597 31166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:11.244999 31166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:11.245121 31166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:11.245241 31166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:11.255412 31166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42299
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:11.256702 31166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:11.258033 31166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:11.272008 31172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:11.272053 31171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:11.272076 31174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:11.273304 31166 server_base.cc:1034] running on GCE node
I20260927 07:06:11.274030 31166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:11.275132 31166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:11.276365 31166 hybrid_clock.cc:648] HybridClock initialized: now 1790492771276325 us; error 58 us; skew 500 ppm
I20260927 07:06:11.276770 31166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:11.280251 31166 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:11.281826 31166 webserver.cc:460] Webserver started at http://127.30.16.67:46207/ using document root <none> and password file <none>
I20260927 07:06:11.282610 31166 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:11.282764 31166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:11.283087 31166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:11.286115 31166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "71d5682ff79e4d37946b2340a50e08c7"
format_stamp: "Formatted at 2026-09-27 07:06:11 on dist-test-slave-8vqf"
server_key: "a72d8ee26d591d0b22d2c3d5f184cd91"
server_key_iv: "9ec38a3870f252fc1379409a0a9b299d"
server_key_version: "encryptionkey@0"
I20260927 07:06:11.286957 31166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "71d5682ff79e4d37946b2340a50e08c7"
format_stamp: "Formatted at 2026-09-27 07:06:11 on dist-test-slave-8vqf"
server_key: "a72d8ee26d591d0b22d2c3d5f184cd91"
server_key_iv: "9ec38a3870f252fc1379409a0a9b299d"
server_key_version: "encryptionkey@0"
I20260927 07:06:11.292330 31166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 07:06:11.295925 31181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:11.297310 31166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:11.297497 31166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "71d5682ff79e4d37946b2340a50e08c7"
format_stamp: "Formatted at 2026-09-27 07:06:11 on dist-test-slave-8vqf"
server_key: "a72d8ee26d591d0b22d2c3d5f184cd91"
server_key_iv: "9ec38a3870f252fc1379409a0a9b299d"
server_key_version: "encryptionkey@0"
I20260927 07:06:11.297706 31166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:11.325309 31166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:11.330940 31166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:11.331321 31166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:11.332551 31166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:11.334360 31166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:11.334468 31166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:11.334638 31166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:11.334719 31166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:11.365105 31026 heartbeater.cc:503] Master 127.30.16.126:42299 was elected leader, sending a full tablet report...
I20260927 07:06:11.370265 31166 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:34521
I20260927 07:06:11.370357 31293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:34521 every 8 connection(s)
I20260927 07:06:11.371809 31166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:11.380712 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31166
I20260927 07:06:11.380968 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:11.381409 30785 external_mini_cluster.cc:1428] Setting key 8d07a4c84773372108f8e9ffdbaee7bb
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:11.392620 31162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.374505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:54811 (local address 127.30.16.126:42299)
0927 07:06:11.374765 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.374772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.375921 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 07:06:11.377798 (+ 1877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.377802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.377875 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.377968 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.379574 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.380429 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.382325 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.382845 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.382984 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.387035 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.387040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.387044 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.387076 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.389669 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.390402 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.390411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.390414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.390605 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.391034 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.391037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.391040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.391314 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.391437 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.392169 (+ 732us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.392378 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 07:06:11.393548 31296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.375053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42299 (local address 127.30.16.67:54811)
0927 07:06:11.375761 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:11.375799 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:11.377531 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:11.378225 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:11.378234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:11.378841 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:11.379412 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:11.379429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.380617 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.380621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:11.382154 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:11.382167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.383091 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.383099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:11.383392 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.384194 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:11.384227 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:11.386845 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:11.389805 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:11.389811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:11.389826 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:11.390235 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:11.390716 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:11.390719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:11.390721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:11.390919 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:11.391419 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:11.391425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:11.391731 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 07:06:11.391956 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 07:06:11.392375 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":288,"threads_started":1}
I20260927 07:06:11.395390 31294 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42299
I20260927 07:06:11.395712 31294 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:11.396565 31294 heartbeater.cc:511] Master 127.30.16.126:42299 requested a full tablet report, sending...
I20260927 07:06:11.398253 30837 ts_manager.cc:194] Registered new tserver with Master: 71d5682ff79e4d37946b2340a50e08c7 (127.30.16.67:34521)
I20260927 07:06:11.399150 30837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:54811
I20260927 07:06:11.410998 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:11.456395 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.440275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.30.16.65:41359)
0927 07:06:11.440980 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.440990 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.441121 (+ 131us) server_negotiation.cc:407] Connection header received
0927 07:06:11.441389 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.441397 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.441628 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.441797 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.443365 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.444373 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.445651 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.446228 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.446372 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.450483 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.450496 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.450513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.450626 (+ 113us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.453792 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.454428 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.454436 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.454438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.454612 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.455013 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.455017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.455019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.455366 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.455612 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.455892 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.456104 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":332,"threads_started":1}
W20260927 07:06:11.457686 30941 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60562
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:11.502981 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.488018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60564 (local address 127.30.16.65:41359)
0927 07:06:11.488270 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.488278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.488538 (+ 260us) server_negotiation.cc:407] Connection header received
0927 07:06:11.488768 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.488773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.488887 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.489000 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.490380 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.491152 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.492411 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.492861 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.492996 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.497285 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.497290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.497293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.497331 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.500314 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.501056 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.501060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.501062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.501226 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.501654 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.501657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.501659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.501924 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.502109 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.502421 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.502731 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 07:06:11.504503 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 30898
I20260927 07:06:11.516966 31300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.505398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60568 (local address 127.30.16.65:41359)
0927 07:06:11.505589 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.505597 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.505916 (+ 319us) server_negotiation.cc:407] Connection header received
0927 07:06:11.506015 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.506018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.506069 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.506144 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.507404 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.508131 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.509391 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.509811 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.509911 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.512179 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.512184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.512188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.512215 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.514817 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.515391 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.515394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.515396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.515514 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.515796 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.515799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.515801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.516032 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.516211 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.516440 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.516584 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:06:11.517707 31021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:11.602849 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 30898
I20260927 07:06:11.618976 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31032
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:11.634882 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.619903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.30.16.66:46509)
0927 07:06:11.620502 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.620509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.620540 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:06:11.620600 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.620606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.620816 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.620949 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.622063 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.623167 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.624365 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.624905 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.625011 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.629128 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.629140 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.629155 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.629194 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.632350 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.632936 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.632942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.632944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.633059 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.633415 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.633417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.633419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.633741 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.633954 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.634235 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.634437 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":324,"threads_started":1}
I20260927 07:06:11.636133 31155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:11.719913 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31032
I20260927 07:06:11.737504 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31166
Sep 27 07:06:11 dist-test-slave-8vqf krb5kdc[30792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:11.753762 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:11.738751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.30.16.67:34521)
0927 07:06:11.739436 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:11.739442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:11.739471 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:06:11.739531 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:11.739537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:11.739749 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:11.739895 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:11.741271 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.742332 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.743759 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:11.744349 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:11.744473 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:11.748598 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:11.748609 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:11.748624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:11.748657 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:11.751415 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.752034 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.752042 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.752044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.752157 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:11.752461 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:11.752464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:11.752467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:11.752801 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:11.753043 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:11.753284 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 07:06:11.753481 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":380,"threads_started":1}
I20260927 07:06:11.755107 31289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:11.838505 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31166
I20260927 07:06:11.854460 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 30805
I20260927 07:06:11.855779 30867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:11.963495 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 30805
2026-09-27T07:06:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3972 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:12 dist-test-slave-8vqf krb5kdc[31315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:12 dist-test-slave-8vqf krb5kdc[31315](info): set up 2 sockets
Sep 27 07:06:12 dist-test-slave-8vqf krb5kdc[31315](info): commencing operation
krb5kdc: starting...
W20260927 07:06:13.046689 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:13 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:13.229921 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44543
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:44543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:13.552312 31328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:13.552610 31328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:13.552778 31328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:13.560056 31328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:13.560216 31328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:13.560297 31328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:13.560395 31328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:13.560472 31328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:13.570520 31328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:44543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:13.571925 31328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:13.573274 31328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:13.585294 31333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:13.585254 31334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:13.585321 31336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:13.586093 31328 server_base.cc:1034] running on GCE node
I20260927 07:06:13.586966 31328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:13.588441 31328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:13.589679 31328 hybrid_clock.cc:648] HybridClock initialized: now 1790492773589647 us; error 57 us; skew 500 ppm
I20260927 07:06:13.590044 31328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:13 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:13.593277 31328 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:13.594964 31328 webserver.cc:460] Webserver started at http://127.30.16.126:45453/ using document root <none> and password file <none>
I20260927 07:06:13.595678 31328 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:13.595825 31328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:13.596143 31328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:13.599184 31328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f"
format_stamp: "Formatted at 2026-09-27 07:06:13 on dist-test-slave-8vqf"
server_key: "2959ee857214236d6b535c469f68422c"
server_key_iv: "803e22be9b167fab870a04de0ad0fbad"
server_key_version: "encryptionkey@0"
I20260927 07:06:13.599936 31328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f"
format_stamp: "Formatted at 2026-09-27 07:06:13 on dist-test-slave-8vqf"
server_key: "2959ee857214236d6b535c469f68422c"
server_key_iv: "803e22be9b167fab870a04de0ad0fbad"
server_key_version: "encryptionkey@0"
I20260927 07:06:13.605304 31328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 07:06:13.608975 31343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:13.610347 31328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:06:13.610632 31328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f"
format_stamp: "Formatted at 2026-09-27 07:06:13 on dist-test-slave-8vqf"
server_key: "2959ee857214236d6b535c469f68422c"
server_key_iv: "803e22be9b167fab870a04de0ad0fbad"
server_key_version: "encryptionkey@0"
I20260927 07:06:13.610850 31328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:13.637871 31328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:13.642279 31328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:13.642628 31328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:13.661465 31328 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:44543
I20260927 07:06:13.661471 31394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:44543 every 8 connection(s)
I20260927 07:06:13.663143 31328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:13.666875 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31328
I20260927 07:06:13.667116 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:13.667248 31395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:13.667814 30785 external_mini_cluster.cc:1428] Setting key 0373c4af583e09474179766cb5426806
I20260927 07:06:13.681523 31395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f: Bootstrap starting.
Sep 27 07:06:13 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:13.685473 31395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:13.686851 31395 log.cc:824] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:13.690438 31395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f: No bootstrap required, opened a new log
I20260927 07:06:13.692126 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:13.669775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.30.16.126:44543)
0927 07:06:13.670639 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:13.670657 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:13.670713 (+ 56us) server_negotiation.cc:407] Connection header received
0927 07:06:13.672361 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:13.672391 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:13.672935 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:13.673410 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:13.674868 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:13.676782 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:13.678025 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:13.679157 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:13.679369 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:13.683248 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:13.683260 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:13.683280 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:13.683330 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:13.687289 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:13.688251 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:13.688260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:13.688301 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:13.688514 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:13.688879 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:13.688884 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:13.688887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:13.689567 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:13.689934 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:13.690233 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:06:13.690768 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":369,"threads_started":1}
I20260927 07:06:13.699003 31395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } }
I20260927 07:06:13.699474 31395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:13.699616 31395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d271a2cab8d46d7a8d6e7dcdf9f8d9f, State: Initialized, Role: FOLLOWER
I20260927 07:06:13.700202 31395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } }
I20260927 07:06:13.700462 31395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:13.700606 31395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:13.700773 31395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:13.703003 31395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } }
I20260927 07:06:13.703524 31395 leader_election.cc:302] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d271a2cab8d46d7a8d6e7dcdf9f8d9f; no voters:
I20260927 07:06:13.704092 31395 leader_election.cc:288] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:13.704231 31400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:13.704898 31400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [term 1 LEADER]: Becoming Leader. State: Replica: 2d271a2cab8d46d7a8d6e7dcdf9f8d9f, State: Running, Role: LEADER
I20260927 07:06:13.705538 31400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } }
I20260927 07:06:13.706303 31395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:13.709158 31401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } } }
I20260927 07:06:13.709532 31401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:13.709897 31402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d271a2cab8d46d7a8d6e7dcdf9f8d9f. Latest consensus state: current_term: 1 leader_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d271a2cab8d46d7a8d6e7dcdf9f8d9f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44543 } } }
I20260927 07:06:13.710266 31402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:13.710810 31408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:13.716813 31408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:13.726066 31408 catalog_manager.cc:1347] Generated new cluster ID: 265c1c4ad608401b96dc222a10117bce
I20260927 07:06:13.726198 31408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:13.741676 31408 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:13.743173 31408 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:13.758138 31408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d271a2cab8d46d7a8d6e7dcdf9f8d9f: Generated new TSK 0
I20260927 07:06:13.759176 31408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:13.799981 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:14.130465 31421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:14.130831 31421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:14.130983 31421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:14.138326 31421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:14.138443 31421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:14.138686 31421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:14.148799 31421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:14.150134 31421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:14.151571 31421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:14.165627 31427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:14.165578 31429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:14.165856 31426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:14.166815 31421 server_base.cc:1034] running on GCE node
I20260927 07:06:14.167520 31421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:14.168573 31421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:14.169838 31421 hybrid_clock.cc:648] HybridClock initialized: now 1790492774169771 us; error 80 us; skew 500 ppm
I20260927 07:06:14.170199 31421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:14 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:14.173622 31421 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:14.175215 31421 webserver.cc:460] Webserver started at http://127.30.16.65:40055/ using document root <none> and password file <none>
I20260927 07:06:14.176014 31421 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:14.176141 31421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:14.176477 31421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:14.179514 31421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "ce4b017b54f4c654640ad35086d3cd63"
server_key_iv: "b868e4a59ca87bb01cf2b3e61bd46e47"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.180295 31421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "ce4b017b54f4c654640ad35086d3cd63"
server_key_iv: "b868e4a59ca87bb01cf2b3e61bd46e47"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.185606 31421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 07:06:14.189182 31436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.190657 31421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:14.190835 31421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "ce4b017b54f4c654640ad35086d3cd63"
server_key_iv: "b868e4a59ca87bb01cf2b3e61bd46e47"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.191044 31421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:14.257174 31421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:14.261519 31421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:14.261821 31421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:14.262830 31421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:14.264315 31421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:14.264418 31421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.264560 31421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:14.264635 31421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.297320 31421 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36615
I20260927 07:06:14.297365 31548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36615 every 8 connection(s)
I20260927 07:06:14.298689 31421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:14.308323 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31421
I20260927 07:06:14.308544 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:14.309330 30785 external_mini_cluster.cc:1428] Setting key e4612b517edeec7e4e20f97aacf9e749
Sep 27 07:06:14 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:14.320894 31551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:14.301253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:44117 (local address 127.30.16.126:44543)
0927 07:06:14.301679 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:14.301684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:14.302771 (+ 1087us) server_negotiation.cc:407] Connection header received
0927 07:06:14.304598 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:14.304602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:14.304667 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:14.304763 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:14.306259 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.307151 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.309375 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.309808 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.309957 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:14.314144 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:14.314149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:14.314152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:14.314184 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:14.317433 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:14.318450 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:14.318454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:14.318456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:14.318624 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:14.319143 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:14.319146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:14.319148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:14.319401 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:14.319514 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:14.320476 (+ 962us) server_negotiation.cc:299] Negotiation successful
0927 07:06:14.320647 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":221,"threads_started":1}
I20260927 07:06:14.322659 31552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:14.301863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:44543 (local address 127.30.16.65:44117)
0927 07:06:14.302601 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:14.302658 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:14.304340 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:14.304967 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:14.304976 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:14.305519 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:14.306110 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:14.306129 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.307325 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.307367 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:14.309234 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:14.309249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.310028 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.310033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:14.310312 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:14.311142 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:14.311177 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:14.313888 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:14.317606 (+ 3718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:14.317619 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:14.317642 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:14.318321 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:14.318741 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:14.318749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:14.318755 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:14.319034 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:14.319538 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:14.319549 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:14.320064 (+ 515us) client_negotiation.cc:769] Sending connection context
0927 07:06:14.320401 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 07:06:14.321026 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":262,"threads_started":1}
I20260927 07:06:14.324312 31549 heartbeater.cc:347] Connected to a master server at 127.30.16.126:44543
I20260927 07:06:14.324688 31549 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:14.325446 31549 heartbeater.cc:511] Master 127.30.16.126:44543 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:14.327641 31360 ts_manager.cc:194] Registered new tserver with Master: 7aa67bace6fe4fc5aa7c90c135b76c3a (127.30.16.65:36615)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:06:14.329241 31360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:44117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:14.351673 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:14.706732 31555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:14.707069 31555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:14.707231 31555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:14.714766 31555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:14.714906 31555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:14.715085 31555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:14.725028 31555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:14.726377 31555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:14.727823 31555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:14.740535 31561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:14.740703 31563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:14.740718 31560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:14.741755 31555 server_base.cc:1034] running on GCE node
I20260927 07:06:14.742337 31555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:14.743355 31555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:14.744562 31555 hybrid_clock.cc:648] HybridClock initialized: now 1790492774744530 us; error 56 us; skew 500 ppm
I20260927 07:06:14.744876 31555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:14 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:14.757761 31555 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:14.759564 31555 webserver.cc:460] Webserver started at http://127.30.16.66:38415/ using document root <none> and password file <none>
I20260927 07:06:14.760512 31555 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:14.760650 31555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:14.760995 31555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:14.764073 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "6d4178362a77453ea6f7c710e6bee42e"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "5d5ba14b42fa144201ca91880e922cac"
server_key_iv: "5c7410cc2930082f6bb6813c466f29b7"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.764869 31555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "6d4178362a77453ea6f7c710e6bee42e"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "5d5ba14b42fa144201ca91880e922cac"
server_key_iv: "5c7410cc2930082f6bb6813c466f29b7"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.770175 31555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 07:06:14.773777 31570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.775213 31555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:06:14.775386 31555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "6d4178362a77453ea6f7c710e6bee42e"
format_stamp: "Formatted at 2026-09-27 07:06:14 on dist-test-slave-8vqf"
server_key: "5d5ba14b42fa144201ca91880e922cac"
server_key_iv: "5c7410cc2930082f6bb6813c466f29b7"
server_key_version: "encryptionkey@0"
I20260927 07:06:14.775599 31555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:14.793536 31555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:14.798128 31555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:14.798441 31555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:14.799507 31555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:14.800953 31555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:14.801052 31555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.801205 31555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:14.801276 31555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:14.836313 31555 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:45259
I20260927 07:06:14.836372 31682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:45259 every 8 connection(s)
I20260927 07:06:14.837638 31555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:14.840073 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31555
I20260927 07:06:14.840271 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:14.840847 30785 external_mini_cluster.cc:1428] Setting key 77718b6168d03e682be0bba224b80686
Sep 27 07:06:14 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:14.860941 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:14.840329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:37935 (local address 127.30.16.126:44543)
0927 07:06:14.840798 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:14.840805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:14.842038 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 07:06:14.843927 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:14.843936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:14.844026 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:14.844150 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:14.845590 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.846852 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.849404 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.849827 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.849928 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:14.854387 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:14.854392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:14.854396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:14.854432 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:14.857338 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:14.858246 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:14.858254 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:14.858259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:14.858398 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:14.858866 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:14.858873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:14.858877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:14.859149 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:14.859424 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:14.860311 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 07:06:14.860550 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:14.862196 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:14.841072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:44543 (local address 127.30.16.66:37935)
0927 07:06:14.841882 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:14.841927 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:14.843659 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:14.844353 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:14.844360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:14.844847 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:14.845418 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:14.845436 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.846998 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.847007 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:14.849254 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:14.849271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:14.850057 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:14.850066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:14.850416 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:14.851166 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:14.851206 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:14.854117 (+ 2911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:14.857516 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:14.857526 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:14.857544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:14.858099 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:14.858514 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:14.858519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:14.858523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:14.858747 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:14.859279 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:14.859287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:14.859889 (+ 602us) client_negotiation.cc:769] Sending connection context
0927 07:06:14.860204 (+ 315us) client_negotiation.cc:240] Negotiation successful
0927 07:06:14.860786 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":331,"threads_started":1}
I20260927 07:06:14.863812 31683 heartbeater.cc:347] Connected to a master server at 127.30.16.126:44543
I20260927 07:06:14.864116 31683 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:14.864837 31683 heartbeater.cc:511] Master 127.30.16.126:44543 requested a full tablet report, sending...
I20260927 07:06:14.866525 31360 ts_manager.cc:194] Registered new tserver with Master: 6d4178362a77453ea6f7c710e6bee42e (127.30.16.66:45259)
I20260927 07:06:14.867415 31360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:37935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:14.885695 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:15.217566 31689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:15.217860 31689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:15.218041 31689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:15.225515 31689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:15.225636 31689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:15.225824 31689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:15.235872 31689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:44543
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:15.237244 31689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:15.238618 31689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:15.251962 31694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:15.251962 31697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:15.252044 31695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:15.252624 31689 server_base.cc:1034] running on GCE node
I20260927 07:06:15.253252 31689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:15.254304 31689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:15.255533 31689 hybrid_clock.cc:648] HybridClock initialized: now 1790492775255469 us; error 55 us; skew 500 ppm
I20260927 07:06:15.255906 31689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:15.259212 31689 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:15.260673 31689 webserver.cc:460] Webserver started at http://127.30.16.67:43045/ using document root <none> and password file <none>
I20260927 07:06:15.261590 31689 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:15.261705 31689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:15.261991 31689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:15.265015 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "ab50f48340dd46eb980500726d2fde65"
format_stamp: "Formatted at 2026-09-27 07:06:15 on dist-test-slave-8vqf"
server_key: "0b132f88f7834a19a79b985d91505e1c"
server_key_iv: "caa3843078d5bd810741fdcef0980425"
server_key_version: "encryptionkey@0"
I20260927 07:06:15.265825 31689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "ab50f48340dd46eb980500726d2fde65"
format_stamp: "Formatted at 2026-09-27 07:06:15 on dist-test-slave-8vqf"
server_key: "0b132f88f7834a19a79b985d91505e1c"
server_key_iv: "caa3843078d5bd810741fdcef0980425"
server_key_version: "encryptionkey@0"
I20260927 07:06:15.271095 31689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 07:06:15.274712 31704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:15.276232 31689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:15.276414 31689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "ab50f48340dd46eb980500726d2fde65"
format_stamp: "Formatted at 2026-09-27 07:06:15 on dist-test-slave-8vqf"
server_key: "0b132f88f7834a19a79b985d91505e1c"
server_key_iv: "caa3843078d5bd810741fdcef0980425"
server_key_version: "encryptionkey@0"
I20260927 07:06:15.276628 31689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:15.307211 31689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:15.311679 31689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:15.312083 31689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:15.313308 31689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:15.314874 31689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:15.314985 31689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:15.315145 31689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:15.315205 31689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:15.332561 31549 heartbeater.cc:503] Master 127.30.16.126:44543 was elected leader, sending a full tablet report...
I20260927 07:06:15.348654 31689 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:38521
I20260927 07:06:15.348697 31816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:38521 every 8 connection(s)
I20260927 07:06:15.349999 31689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:15.354163 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31689
I20260927 07:06:15.354393 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:15.354946 30785 external_mini_cluster.cc:1428] Setting key 213905a2dda960338db1b277bb7a7436
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:15.371033 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.352766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45421 (local address 127.30.16.126:44543)
0927 07:06:15.352955 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.352959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.354446 (+ 1487us) server_negotiation.cc:407] Connection header received
0927 07:06:15.356672 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.356676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.356781 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.356892 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.358723 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.359439 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.361018 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.361587 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.361683 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.365151 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.365155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.365159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.365193 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.367887 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.368656 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.368660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.368663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.368829 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.369270 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.369282 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.369285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.369574 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.369734 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.370454 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.370702 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:06:15.372047 31819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.353355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:44543 (local address 127.30.16.67:45421)
0927 07:06:15.354234 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.354296 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.356364 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.357216 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.357225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.357840 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:15.358525 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.358586 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.359591 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.359600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.360833 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.360901 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.361792 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.361797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.362004 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.362692 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:15.362714 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:15.364917 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:15.368034 (+ 3117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:15.368044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:15.368062 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:15.368519 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:15.368960 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:15.368964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:15.368966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:15.369142 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:15.369709 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:15.369718 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:15.370074 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.370338 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.370923 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":326,"threads_started":1}
I20260927 07:06:15.373335 31817 heartbeater.cc:347] Connected to a master server at 127.30.16.126:44543
I20260927 07:06:15.373665 31817 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:15.374370 31817 heartbeater.cc:511] Master 127.30.16.126:44543 requested a full tablet report, sending...
I20260927 07:06:15.375809 31360 ts_manager.cc:194] Registered new tserver with Master: ab50f48340dd46eb980500726d2fde65 (127.30.16.67:38521)
I20260927 07:06:15.376600 31360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:45421
I20260927 07:06:15.382639 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:15.401582 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.388903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.30.16.126:44543)
0927 07:06:15.389072 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.389077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.389477 (+ 400us) server_negotiation.cc:407] Connection header received
0927 07:06:15.389618 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.389621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.389692 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.389776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.391268 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.391986 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.393272 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.393675 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.393769 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.396252 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.396256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.396260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.396292 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.399236 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.399889 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.399893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.399895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.400024 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.400347 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.400351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.400353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.400666 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.400861 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.401184 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.401346 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:06:15.408452 31360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:15.415588 31360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:15.441978 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.427660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36615 (local address 127.0.0.1:34104)
0927 07:06:15.430019 (+ 2359us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.430080 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.430286 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.431784 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.431792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.431845 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:15.432261 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.432273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.434990 (+ 2717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.434994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.438116 (+ 3122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.438142 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.440470 (+ 2328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.440480 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.440795 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.440876 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.441020 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.441292 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2023,"spinlock_wait_cycles":2816,"thread_start_us":3769,"threads_started":1}
I20260927 07:06:15.443415 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.427843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:45259 (local address 127.0.0.1:56102)
0927 07:06:15.431689 (+ 3846us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.431714 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.431860 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.432903 (+ 1043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.432911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.432945 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:15.433471 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.433487 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.436001 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.436009 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.439188 (+ 3179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.439209 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.442433 (+ 3224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.442446 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.442712 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.442753 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.442878 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.443010 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3715,"spinlock_wait_cycles":9216,"thread_start_us":1686,"threads_started":1}
I20260927 07:06:15.443410 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.428392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.30.16.66:45259)
0927 07:06:15.431295 (+ 2903us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.431303 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.432002 (+ 699us) server_negotiation.cc:407] Connection header received
0927 07:06:15.432109 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.432116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.432390 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.432618 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:15.433706 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.435802 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.439443 (+ 3641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.441514 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.441689 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.441966 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.442931 (+ 965us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.443103 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2763,"thread_start_us":2638,"threads_started":1}
I20260927 07:06:15.443720 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.427990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.30.16.65:36615)
0927 07:06:15.431032 (+ 3042us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.431044 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.431074 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:06:15.431177 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.431186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.431525 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.432465 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:15.432724 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.434809 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.438412 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.440183 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.442738 (+ 2555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.442981 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.443056 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.443217 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2860,"thread_start_us":2683,"threads_started":1}
I20260927 07:06:15.447363 31618 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f27fbe7b40cb9011395a94f82118 (DEFAULT_TABLE table=test-table [id=c1ca9d87eead440dbe950619f5a9418e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:15.447582 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f27fbe7b40cb9011395a94f82118 (DEFAULT_TABLE table=test-table [id=c1ca9d87eead440dbe950619f5a9418e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:15.448728 31618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f27fbe7b40cb9011395a94f82118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:15.448946 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f27fbe7b40cb9011395a94f82118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:15.450779 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.432620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:38521 (local address 127.0.0.1:58526)
0927 07:06:15.442368 (+ 9748us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.442387 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.442532 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.443679 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.443685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.443721 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:15.443877 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.443889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.446145 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.446150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.447959 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.447984 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.449739 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.449746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.450156 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.450196 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.450296 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.450440 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9643,"thread_start_us":9955,"threads_started":1}
I20260927 07:06:15.451673 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.432081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58526 (local address 127.30.16.67:38521)
0927 07:06:15.442528 (+ 10447us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.442537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.443047 (+ 510us) server_negotiation.cc:407] Connection header received
0927 07:06:15.443151 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.443161 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.443447 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.444098 (+ 651us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:15.444361 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.445975 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.448185 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.449572 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.449703 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.449924 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.451169 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.451327 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10243,"thread_start_us":10091,"threads_started":1}
I20260927 07:06:15.455477 31752 tablet_service.cc:1467] Processing CreateTablet for tablet 0f25f27fbe7b40cb9011395a94f82118 (DEFAULT_TABLE table=test-table [id=c1ca9d87eead440dbe950619f5a9418e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:15.456970 31752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f25f27fbe7b40cb9011395a94f82118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:15.460493 31835 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Bootstrap starting.
I20260927 07:06:15.462842 31836 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Bootstrap starting.
I20260927 07:06:15.464184 31835 tablet_bootstrap.cc:654] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:15.465458 31835 log.cc:824] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:15.466387 31836 tablet_bootstrap.cc:654] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:15.467679 31836 log.cc:824] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:15.469692 31835 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: No bootstrap required, opened a new log
I20260927 07:06:15.469996 31835 ts_tablet_manager.cc:1397] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 07:06:15.470676 31836 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: No bootstrap required, opened a new log
I20260927 07:06:15.470927 31836 ts_tablet_manager.cc:1397] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20260927 07:06:15.471362 31839 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Bootstrap starting.
I20260927 07:06:15.474143 31839 tablet_bootstrap.cc:654] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:15.475170 31839 log.cc:824] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:15.478087 31839 tablet_bootstrap.cc:492] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: No bootstrap required, opened a new log
I20260927 07:06:15.478349 31839 ts_tablet_manager.cc:1397] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260927 07:06:15.480652 31835 raft_consensus.cc:348] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.481117 31835 raft_consensus.cc:374] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:15.481217 31835 raft_consensus.cc:729] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4178362a77453ea6f7c710e6bee42e, State: Initialized, Role: FOLLOWER
I20260927 07:06:15.481374 31836 raft_consensus.cc:348] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.481779 31836 raft_consensus.cc:374] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:15.481879 31836 raft_consensus.cc:729] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa67bace6fe4fc5aa7c90c135b76c3a, State: Initialized, Role: FOLLOWER
I20260927 07:06:15.481985 31835 consensus_queue.cc:260] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.482481 31836 consensus_queue.cc:260] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.483283 31683 heartbeater.cc:503] Master 127.30.16.126:44543 was elected leader, sending a full tablet report...
I20260927 07:06:15.484014 31835 ts_tablet_manager.cc:1428] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260927 07:06:15.485858 31836 ts_tablet_manager.cc:1428] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Time spent starting tablet: real 0.015s user 0.012s sys 0.002s
I20260927 07:06:15.488793 31839 raft_consensus.cc:348] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.489243 31839 raft_consensus.cc:374] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:15.489343 31839 raft_consensus.cc:729] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab50f48340dd46eb980500726d2fde65, State: Initialized, Role: FOLLOWER
I20260927 07:06:15.489955 31839 consensus_queue.cc:260] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.491114 31817 heartbeater.cc:503] Master 127.30.16.126:44543 was elected leader, sending a full tablet report...
I20260927 07:06:15.491699 31839 ts_tablet_manager.cc:1428] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260927 07:06:15.520259 31841 raft_consensus.cc:482] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:15.520589 31841 raft_consensus.cc:504] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.522262 31841 leader_election.cc:288] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aa67bace6fe4fc5aa7c90c135b76c3a (127.30.16.65:36615), ab50f48340dd46eb980500726d2fde65 (127.30.16.67:38521)
I20260927 07:06:15.529105 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.522873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43331 (local address 127.30.16.65:36615)
0927 07:06:15.523077 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.523082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.523478 (+ 396us) server_negotiation.cc:407] Connection header received
0927 07:06:15.523705 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.523712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.523797 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.523908 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:15.524537 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.525224 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.527451 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.528186 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.528381 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.528469 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.528729 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.528860 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:06:15.529191 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.522697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36615 (local address 127.30.16.66:43331)
0927 07:06:15.523352 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.523379 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.523525 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.524035 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.524042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.524074 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:15.524389 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.524398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.525357 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.525364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.527291 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.527303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.528404 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.528410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.528572 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.528595 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.528704 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.528827 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":324,"threads_started":1}
I20260927 07:06:15.530025 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.523489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:38521 (local address 127.30.16.66:52395)
0927 07:06:15.524197 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:15.524211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:15.524364 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:15.524698 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:15.524702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:15.524732 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:15.525034 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.525042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.526154 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.526161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.527928 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:15.527940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.529102 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.529108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:15.529495 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.529532 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:06:15.529635 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:06:15.529758 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":426,"threads_started":1}
I20260927 07:06:15.530063 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.523733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52395 (local address 127.30.16.67:38521)
0927 07:06:15.523939 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.523944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.524311 (+ 367us) server_negotiation.cc:407] Connection header received
0927 07:06:15.524512 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.524517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.524584 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.524682 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:15.525195 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.526018 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.528100 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.528883 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.529074 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.529148 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.529672 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.529791 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:06:15.530346 31504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f27fbe7b40cb9011395a94f82118" candidate_uuid: "6d4178362a77453ea6f7c710e6bee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" is_pre_election: true
I20260927 07:06:15.530934 31504 raft_consensus.cc:2454] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4178362a77453ea6f7c710e6bee42e in term 0.
I20260927 07:06:15.531026 31772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f27fbe7b40cb9011395a94f82118" candidate_uuid: "6d4178362a77453ea6f7c710e6bee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab50f48340dd46eb980500726d2fde65" is_pre_election: true
I20260927 07:06:15.531558 31772 raft_consensus.cc:2454] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d4178362a77453ea6f7c710e6bee42e in term 0.
I20260927 07:06:15.531600 31571 leader_election.cc:302] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4178362a77453ea6f7c710e6bee42e, 7aa67bace6fe4fc5aa7c90c135b76c3a; no voters:
I20260927 07:06:15.532028 31841 raft_consensus.cc:2781] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:15.532177 31841 raft_consensus.cc:482] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:15.532284 31841 raft_consensus.cc:3037] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:15.534457 31841 raft_consensus.cc:504] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.535130 31841 leader_election.cc:288] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [CANDIDATE]: Term 1 election: Requested vote from peers 7aa67bace6fe4fc5aa7c90c135b76c3a (127.30.16.65:36615), ab50f48340dd46eb980500726d2fde65 (127.30.16.67:38521)
I20260927 07:06:15.535528 31504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f27fbe7b40cb9011395a94f82118" candidate_uuid: "6d4178362a77453ea6f7c710e6bee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a"
I20260927 07:06:15.535621 31772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f25f27fbe7b40cb9011395a94f82118" candidate_uuid: "6d4178362a77453ea6f7c710e6bee42e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab50f48340dd46eb980500726d2fde65"
I20260927 07:06:15.535732 31504 raft_consensus.cc:3037] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:15.535856 31772 raft_consensus.cc:3037] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:15.538029 31772 raft_consensus.cc:2454] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4178362a77453ea6f7c710e6bee42e in term 1.
I20260927 07:06:15.538092 31504 raft_consensus.cc:2454] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d4178362a77453ea6f7c710e6bee42e in term 1.
I20260927 07:06:15.538441 31571 leader_election.cc:302] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4178362a77453ea6f7c710e6bee42e, ab50f48340dd46eb980500726d2fde65; no voters:
I20260927 07:06:15.538769 31841 raft_consensus.cc:2781] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:15.539199 31841 raft_consensus.cc:686] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [term 1 LEADER]: Becoming Leader. State: Replica: 6d4178362a77453ea6f7c710e6bee42e, State: Running, Role: LEADER
I20260927 07:06:15.539680 31841 consensus_queue.cc:237] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } }
I20260927 07:06:15.544538 31360 catalog_manager.cc:5495] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d4178362a77453ea6f7c710e6bee42e (127.30.16.66). New cstate: current_term: 1 leader_uuid: "6d4178362a77453ea6f7c710e6bee42e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aa67bace6fe4fc5aa7c90c135b76c3a" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d4178362a77453ea6f7c710e6bee42e" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab50f48340dd46eb980500726d2fde65" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38521 } health_report { overall_health: UNKNOWN } } }
W20260927 07:06:15.551198 31550 tablet.cc:2369] T 0f25f27fbe7b40cb9011395a94f82118 P 7aa67bace6fe4fc5aa7c90c135b76c3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:15.567507 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.553158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34110 (local address 127.30.16.65:36615)
0927 07:06:15.553314 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.553318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.553860 (+ 542us) server_negotiation.cc:407] Connection header received
0927 07:06:15.553949 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.553953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.554015 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.554096 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.555586 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.556359 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.557559 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.558007 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.558105 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.562204 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.562220 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.562238 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.562272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.565202 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.565801 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.565808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.565810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.565971 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.566267 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.566270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.566272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.566631 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.566838 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.567078 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.567231 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 07:06:15.589797 31684 tablet.cc:2369] T 0f25f27fbe7b40cb9011395a94f82118 P 6d4178362a77453ea6f7c710e6bee42e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 07:06:15.601696 31818 tablet.cc:2369] T 0f25f27fbe7b40cb9011395a94f82118 P ab50f48340dd46eb980500726d2fde65: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:15.613723 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.599693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34120 (local address 127.30.16.65:36615)
0927 07:06:15.599969 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.599975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.600177 (+ 202us) server_negotiation.cc:407] Connection header received
0927 07:06:15.600327 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.600331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.600397 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.600486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.601784 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.602655 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.603918 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.604414 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.604546 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.608463 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.608468 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.608471 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.608505 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.611188 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.611914 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.611918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.611920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.612044 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.612414 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.612417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.612419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.612733 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.612985 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.613212 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.613386 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:15.657863 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.643647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34132 (local address 127.30.16.65:36615)
0927 07:06:15.643890 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.643899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.644218 (+ 319us) server_negotiation.cc:407] Connection header received
0927 07:06:15.644436 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.644441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.644504 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.644613 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.646058 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.646820 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.648047 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.648581 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.648718 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.652664 (+ 3946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.652669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.652672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.652708 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.655239 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.655887 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.655895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.655898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.656022 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.656378 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.656383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.656387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.656711 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.656971 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.657276 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.657518 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 07:06:15.659737 31464 tablet_service.cc:2908] Rejecting scan request for tablet 0f25f27fbe7b40cb9011395a94f82118: Uninitialized: clean time has not yet been initialized
I20260927 07:06:15.662990 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31421
I20260927 07:06:15.677394 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.663745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.30.16.65:36615)
0927 07:06:15.663942 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.663948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.664380 (+ 432us) server_negotiation.cc:407] Connection header received
0927 07:06:15.664546 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.664550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.664620 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.664694 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.666200 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.667072 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.668823 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.669204 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.669306 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.672034 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.672038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.672042 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.672071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.674716 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.675517 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.675521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.675524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.675664 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.676062 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.676066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.676068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.676340 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.676547 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.676868 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.677034 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 07:06:15.678118 31544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:15.768024 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31421
I20260927 07:06:15.785530 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31555
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:15.800920 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.786325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.30.16.66:45259)
0927 07:06:15.786528 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.786536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.786574 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:06:15.786774 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.786786 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.786853 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.786983 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.788108 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.788830 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.790082 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.790493 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.790675 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.795207 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.795223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.795240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.795274 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.798261 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.799015 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.799023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.799026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.799158 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.799530 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.799533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.799535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.799883 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.800106 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.800360 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.800553 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:06:15.801656 31678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:15.891808 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31555
I20260927 07:06:15.909013 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31689
Sep 27 07:06:15 dist-test-slave-8vqf krb5kdc[31315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:15.924099 31833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:15.909756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58534 (local address 127.30.16.67:38521)
0927 07:06:15.909996 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:15.910002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:15.910021 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:06:15.910217 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:15.910221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:15.910284 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:15.910393 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:15.911767 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.912522 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.913823 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:15.914360 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:15.914497 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:15.918673 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:15.918688 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:15.918704 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:15.918744 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:15.921366 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.922031 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.922049 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.922051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.922173 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:15.922588 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:15.922593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:15.922599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:15.922967 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:15.923254 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:15.923541 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 07:06:15.923755 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:06:15.924779 31812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:16.013558 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31689
I20260927 07:06:16.030230 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31328
I20260927 07:06:16.031515 31390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:16.140491 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31328
2026-09-27T07:06:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4177 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:16 dist-test-slave-8vqf krb5kdc[31865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:16 dist-test-slave-8vqf krb5kdc[31865](info): set up 2 sockets
Sep 27 07:06:16 dist-test-slave-8vqf krb5kdc[31865](info): commencing operation
krb5kdc: starting...
W20260927 07:06:17.226997 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:17 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:17.415134 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39301
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:17.740007 31878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:17.740316 31878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:17.740481 31878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:17.748116 31878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:17.748234 31878 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:06:17.748275 31878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:17.748400 31878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:17.748467 31878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:17.748549 31878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:17.758796 31878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39301
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:17.760180 31878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:17.761585 31878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:17.774048 31884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:17.774021 31886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:17.774212 31883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:17.774741 31878 server_base.cc:1034] running on GCE node
I20260927 07:06:17.775676 31878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:17.777585 31878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:17.778838 31878 hybrid_clock.cc:648] HybridClock initialized: now 1790492777778781 us; error 47 us; skew 500 ppm
I20260927 07:06:17.779219 31878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:17 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:17.782541 31878 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:17.784214 31878 webserver.cc:460] Webserver started at http://127.30.16.126:37353/ using document root <none> and password file <none>
I20260927 07:06:17.784996 31878 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:17.785084 31878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:17.785419 31878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:17.788552 31878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "f71d67987d9f4a93b416fd4c4f80d625"
format_stamp: "Formatted at 2026-09-27 07:06:17 on dist-test-slave-8vqf"
server_key: "3232a34bd6c313c1e1890db0c2d05d25"
server_key_iv: "6b1e4a553bfabbc3d4b001923a7c6169"
server_key_version: "encryptionkey@0"
I20260927 07:06:17.789378 31878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "f71d67987d9f4a93b416fd4c4f80d625"
format_stamp: "Formatted at 2026-09-27 07:06:17 on dist-test-slave-8vqf"
server_key: "3232a34bd6c313c1e1890db0c2d05d25"
server_key_iv: "6b1e4a553bfabbc3d4b001923a7c6169"
server_key_version: "encryptionkey@0"
I20260927 07:06:17.795233 31878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:17.799213 31893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:17.800675 31878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:17.800868 31878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "f71d67987d9f4a93b416fd4c4f80d625"
format_stamp: "Formatted at 2026-09-27 07:06:17 on dist-test-slave-8vqf"
server_key: "3232a34bd6c313c1e1890db0c2d05d25"
server_key_iv: "6b1e4a553bfabbc3d4b001923a7c6169"
server_key_version: "encryptionkey@0"
I20260927 07:06:17.801111 31878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:17.820812 31878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:17.825470 31878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:17.825796 31878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:17.848110 31878 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:39301
I20260927 07:06:17.848142 31954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:39301 every 8 connection(s)
I20260927 07:06:17.849546 31878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:17.853341 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31878
I20260927 07:06:17.853580 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:17.853741 31955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:17.854429 30785 external_mini_cluster.cc:1428] Setting key 18188961fce939ebcba3279ae8fa770f
I20260927 07:06:17.869030 31955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Bootstrap starting.
Sep 27 07:06:17 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:17.872387 31955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:17.873667 31955 log.cc:824] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:17.876612 31955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: No bootstrap required, opened a new log
I20260927 07:06:17.877951 31958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:17.856558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.30.16.126:39301)
0927 07:06:17.857470 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:17.857481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:17.857528 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:06:17.859152 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:17.859174 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:17.859586 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:17.859928 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:17.861646 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:17.863039 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:17.864776 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:17.865564 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:17.865722 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:17.870612 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:17.870631 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:17.870646 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:17.870686 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:17.873847 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:17.874581 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:17.874590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:17.874599 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:17.874780 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:17.875170 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:17.875175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:17.875179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:17.875712 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:17.876017 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:17.876446 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 07:06:17.876905 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":467,"threads_started":1}
I20260927 07:06:17.884464 31955 raft_consensus.cc:348] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } }
I20260927 07:06:17.884902 31955 raft_consensus.cc:374] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:17.885017 31955 raft_consensus.cc:729] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71d67987d9f4a93b416fd4c4f80d625, State: Initialized, Role: FOLLOWER
I20260927 07:06:17.885658 31955 consensus_queue.cc:260] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } }
I20260927 07:06:17.885917 31955 raft_consensus.cc:388] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:17.886058 31955 raft_consensus.cc:482] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:17.886243 31955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:17.888617 31955 raft_consensus.cc:504] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } }
I20260927 07:06:17.889194 31955 leader_election.cc:302] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f71d67987d9f4a93b416fd4c4f80d625; no voters:
I20260927 07:06:17.889849 31955 leader_election.cc:288] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:17.889997 31960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:17.890823 31960 raft_consensus.cc:686] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [term 1 LEADER]: Becoming Leader. State: Replica: f71d67987d9f4a93b416fd4c4f80d625, State: Running, Role: LEADER
I20260927 07:06:17.891415 31960 consensus_queue.cc:237] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } }
I20260927 07:06:17.892270 31955 sys_catalog.cc:564] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:17.894472 31962 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f71d67987d9f4a93b416fd4c4f80d625. Latest consensus state: current_term: 1 leader_uuid: "f71d67987d9f4a93b416fd4c4f80d625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } } }
I20260927 07:06:17.895229 31962 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:17.895043 31961 sys_catalog.cc:455] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f71d67987d9f4a93b416fd4c4f80d625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f71d67987d9f4a93b416fd4c4f80d625" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39301 } } }
I20260927 07:06:17.895358 31961 sys_catalog.cc:458] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:17.899749 31968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:17.904814 31968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:17.913329 31968 catalog_manager.cc:1347] Generated new cluster ID: 1e0c3e8816be41c9b482c59c0a7ea7e7
I20260927 07:06:17.913463 31968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:17.932905 31968 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:17.934297 31968 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:17.948778 31968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Generated new TSK 0
I20260927 07:06:17.949721 31968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:17.999449 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39301
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:18.328899 31981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:18.329247 31981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:18.329432 31981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:18.336798 31981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:18.336973 31981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:18.337142 31981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:18.337289 31981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:18.347754 31981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:18.349205 31981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:18.350639 31981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:18.363708 31986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:18.363739 31987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:18.363952 31989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:18.364720 31981 server_base.cc:1034] running on GCE node
I20260927 07:06:18.365288 31981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:18.366255 31981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:18.367470 31981 hybrid_clock.cc:648] HybridClock initialized: now 1790492778367373 us; error 67 us; skew 500 ppm
I20260927 07:06:18.367744 31981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:18 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492778, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:18.370996 31981 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:18.372524 31981 webserver.cc:460] Webserver started at http://127.30.16.65:37745/ using document root <none> and password file <none>
I20260927 07:06:18.373291 31981 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:18.373425 31981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:18.373757 31981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:18.376971 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "01e118767c524f15aa5932da4a840bf2"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "5d9363235aedaa60809392dfa95ee03c"
server_key_iv: "013b38b078a057fbdce354d17982c972"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.377774 31981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "01e118767c524f15aa5932da4a840bf2"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "5d9363235aedaa60809392dfa95ee03c"
server_key_iv: "013b38b078a057fbdce354d17982c972"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.383514 31981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 07:06:18.387027 31996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:18.388401 31981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:06:18.388586 31981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "01e118767c524f15aa5932da4a840bf2"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "5d9363235aedaa60809392dfa95ee03c"
server_key_iv: "013b38b078a057fbdce354d17982c972"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.388801 31981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:18.413739 31981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:18.418027 31981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:18.418321 31981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:18.420960 31981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:18.421078 31981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:18.421254 31981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:18.421332 31981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:06:18 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492778, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:18.446236 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.421851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:50917 (local address 127.30.16.126:39301)
0927 07:06:18.422486 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:18.422493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:18.423747 (+ 1254us) server_negotiation.cc:407] Connection header received
0927 07:06:18.426864 (+ 3117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:18.426873 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:18.426979 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:18.427118 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:18.429965 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.430969 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.434514 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.435122 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.435282 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.440247 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:18.440252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:18.440256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:18.440288 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:18.443159 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.443906 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.443915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.443921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.444039 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.444483 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.444491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.444496 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.444763 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:18.444916 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:18.445632 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 07:06:18.445842 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":372,"threads_started":1}
I20260927 07:06:18.447932 32006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.422593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.65:50917)
0927 07:06:18.423552 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:18.423610 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:18.426445 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:18.427468 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:18.427488 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:18.428550 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:18.429710 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.429752 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.431149 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.431161 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.434304 (+ 3143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.434326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.435447 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.435455 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.435688 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.436507 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:18.436553 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:18.439957 (+ 3404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:18.443312 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.443321 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.443340 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.443769 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.444160 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.444165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.444168 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.444363 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.444899 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:18.444908 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:18.445277 (+ 369us) client_negotiation.cc:769] Sending connection context
0927 07:06:18.445562 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 07:06:18.446107 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":347,"threads_started":1}
I20260927 07:06:18.472272 32077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.456229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.65:44793)
0927 07:06:18.456846 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:18.456871 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:18.457035 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:18.457478 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:18.457485 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:18.458239 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:18.458687 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.458708 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.460371 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.460377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.462412 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.462426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.463703 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.463719 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.463837 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.464488 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:18.464518 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:18.465335 (+ 817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:18.468997 (+ 3662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.469000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.469003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.469761 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.470262 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.470269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.470279 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.470483 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.471149 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:18.471153 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:18.471312 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 07:06:18.471428 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 07:06:18.471754 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":314,"threads_started":1}
I20260927 07:06:18.472308 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.456718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:44793 (local address 127.30.16.126:39301)
0927 07:06:18.456951 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:18.456957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:18.456978 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:06:18.457201 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:18.457211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:18.457307 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:18.457423 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:18.458864 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.460167 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.462621 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.463295 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.463464 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.465529 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:18.465538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:18.465545 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:18.465590 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:18.468876 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.469907 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.469914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.469921 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.470118 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.470639 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.470647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.470654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.471030 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:18.471188 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:18.471571 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 07:06:18.471848 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:06:18.473201 31981 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:38067
I20260927 07:06:18.473249 32116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:38067 every 8 connection(s)
I20260927 07:06:18.474663 31981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:18.476718 32117 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39301
I20260927 07:06:18.477283 32117 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:18.477908 32117 heartbeater.cc:511] Master 127.30.16.126:39301 requested a full tablet report, sending...
I20260927 07:06:18.478670 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 31981
I20260927 07:06:18.478914 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:18.479523 30785 external_mini_cluster.cc:1428] Setting key 77b9490970c7804aaab9b8f58374ca16
I20260927 07:06:18.479930 31910 ts_manager.cc:194] Registered new tserver with Master: 01e118767c524f15aa5932da4a840bf2 (127.30.16.65:38067)
I20260927 07:06:18.481760 31910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:50917
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:18.523029 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39301
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:18.858713 32121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:18.859097 32121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:18.859239 32121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:18.870980 32121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:18.871127 32121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:18.871258 32121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:18.871384 32121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:18.881623 32121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:18.883039 32121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:18.884368 32121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:18.898394 32127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:18.898422 32126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:18.898458 32129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:18.899241 32121 server_base.cc:1034] running on GCE node
I20260927 07:06:18.899827 32121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:18.900894 32121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:18.902117 32121 hybrid_clock.cc:648] HybridClock initialized: now 1790492778902037 us; error 63 us; skew 500 ppm
I20260927 07:06:18.902385 32121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:18 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492778, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:18.905802 32121 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:18.907326 32121 webserver.cc:460] Webserver started at http://127.30.16.66:44011/ using document root <none> and password file <none>
I20260927 07:06:18.908001 32121 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:18.908088 32121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:18.908380 32121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:18.911535 32121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "87c5964bd6dc423a9a4663d1a30884c9"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "ac003f1ab4a81c6577d9bfbf50dbfbd2"
server_key_iv: "8e2cb263ddd58faf87b7586b42237e60"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.912295 32121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "87c5964bd6dc423a9a4663d1a30884c9"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "ac003f1ab4a81c6577d9bfbf50dbfbd2"
server_key_iv: "8e2cb263ddd58faf87b7586b42237e60"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.917444 32121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:06:18.921082 32136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:18.922667 32121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:18.922851 32121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "87c5964bd6dc423a9a4663d1a30884c9"
format_stamp: "Formatted at 2026-09-27 07:06:18 on dist-test-slave-8vqf"
server_key: "ac003f1ab4a81c6577d9bfbf50dbfbd2"
server_key_iv: "8e2cb263ddd58faf87b7586b42237e60"
server_key_version: "encryptionkey@0"
I20260927 07:06:18.923077 32121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:18.947916 32121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:18.952713 32121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:18.952963 32121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:18.955780 32121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:18.955863 32121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:18.956053 32121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:18.956136 32121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:06:18 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492778, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:18.982956 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.956277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:48491 (local address 127.30.16.126:39301)
0927 07:06:18.956691 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:18.956698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:18.958146 (+ 1448us) server_negotiation.cc:407] Connection header received
0927 07:06:18.960298 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:18.960304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:18.960400 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:18.960530 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:18.963140 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.964260 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.966899 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.967553 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.967707 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.973452 (+ 5745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:18.973463 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:18.973468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:18.973513 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:18.977613 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.978697 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.978705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.978711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.978894 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:18.979586 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:18.979592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:18.979597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:18.980046 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:18.980209 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:18.982326 (+ 2117us) server_negotiation.cc:299] Negotiation successful
0927 07:06:18.982593 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260927 07:06:18.983692 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.957057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.66:48491)
0927 07:06:18.957949 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:18.958004 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:18.959993 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:18.960882 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:18.960898 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:18.961799 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:18.962851 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.962890 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.964442 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.964448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.966754 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.966766 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.967846 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.967854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.968086 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.969049 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:18.969091 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:18.973059 (+ 3968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:18.977795 (+ 4736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.977810 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.977838 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.978521 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.979046 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:18.979054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:18.979060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:18.979422 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:18.980304 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:18.980316 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:18.981033 (+ 717us) client_negotiation.cc:769] Sending connection context
0927 07:06:18.981466 (+ 433us) client_negotiation.cc:240] Negotiation successful
0927 07:06:18.982235 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":366,"threads_started":1}
I20260927 07:06:19.001955 32121 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:32805
I20260927 07:06:19.002046 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:32805 every 8 connection(s)
I20260927 07:06:19.003719 32121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:19.005621 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.989547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52345 (local address 127.30.16.126:39301)
0927 07:06:18.989773 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:18.989781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:18.990465 (+ 684us) server_negotiation.cc:407] Connection header received
0927 07:06:18.990829 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:18.990838 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:18.990902 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:18.991017 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:18.992427 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.993321 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.995488 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.995894 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.996042 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.998934 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:18.998942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:18.998950 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:18.998999 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:19.002381 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.003200 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.003204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.003206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.003333 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.003724 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.003727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.003729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.004110 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:19.004277 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.005021 (+ 744us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.005284 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:06:19.005628 32226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:18.989626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.66:52345)
0927 07:06:18.990345 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:18.990370 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:18.990657 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:18.991052 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:18.991057 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:18.991773 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:18.992262 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.992276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.993513 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.993518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.995317 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:18.995328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:18.996087 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:18.996093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:18.996349 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:18.997160 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:18.997221 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:18.998707 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:19.002519 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.002524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.002528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.003077 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.003451 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.003455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.003459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.003612 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.004264 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:19.004269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:19.004596 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.004838 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.005186 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":338,"threads_started":1}
I20260927 07:06:19.006204 32256 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39301
I20260927 07:06:19.006453 32256 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:19.007134 32256 heartbeater.cc:511] Master 127.30.16.126:39301 requested a full tablet report, sending...
I20260927 07:06:19.008652 31910 ts_manager.cc:194] Registered new tserver with Master: 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805)
I20260927 07:06:19.009778 31910 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:48491
I20260927 07:06:19.011920 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 32121
I20260927 07:06:19.012122 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:19.012555 30785 external_mini_cluster.cc:1428] Setting key 862a15309e82364f5df395957af1d1f8
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:19.054052 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39301
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:19.383677 32260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:19.383998 32260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:19.384096 32260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:19.391692 32260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:19.391829 32260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:19.392011 32260 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:19.392158 32260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:19.402072 32260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:19.403491 32260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:19.404814 32260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:19.418942 32265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:19.418875 32268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:19.418915 32266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:19.419951 32260 server_base.cc:1034] running on GCE node
I20260927 07:06:19.420641 32260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:19.421640 32260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:19.422910 32260 hybrid_clock.cc:648] HybridClock initialized: now 1790492779422840 us; error 61 us; skew 500 ppm
I20260927 07:06:19.423240 32260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:19 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:19.426765 32260 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:19.428598 32260 webserver.cc:460] Webserver started at http://127.30.16.67:42155/ using document root <none> and password file <none>
I20260927 07:06:19.429382 32260 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:19.429481 32260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:19.429881 32260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:19.433154 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "f516a174a7de4d60818610e13399d368"
format_stamp: "Formatted at 2026-09-27 07:06:19 on dist-test-slave-8vqf"
server_key: "54ddc1f5c6f86a6d55766497eb898d9b"
server_key_iv: "b24b7ffab6dcda5a79e78b8ff6149fec"
server_key_version: "encryptionkey@0"
I20260927 07:06:19.433981 32260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "f516a174a7de4d60818610e13399d368"
format_stamp: "Formatted at 2026-09-27 07:06:19 on dist-test-slave-8vqf"
server_key: "54ddc1f5c6f86a6d55766497eb898d9b"
server_key_iv: "b24b7ffab6dcda5a79e78b8ff6149fec"
server_key_version: "encryptionkey@0"
I20260927 07:06:19.439477 32260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:19.443212 32275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:19.444854 32260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:06:19.445052 32260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "f516a174a7de4d60818610e13399d368"
format_stamp: "Formatted at 2026-09-27 07:06:19 on dist-test-slave-8vqf"
server_key: "54ddc1f5c6f86a6d55766497eb898d9b"
server_key_iv: "b24b7ffab6dcda5a79e78b8ff6149fec"
server_key_version: "encryptionkey@0"
I20260927 07:06:19.445264 32260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:19.466820 32260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:19.471349 32260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:19.471652 32260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:19.474650 32260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:19.474758 32260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:19.474920 32260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:19.475003 32260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:19.484542 32117 heartbeater.cc:503] Master 127.30.16.126:39301 was elected leader, sending a full tablet report...
Sep 27 07:06:19 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:19.503814 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.475635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45621 (local address 127.30.16.126:39301)
0927 07:06:19.475861 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.475869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.477826 (+ 1957us) server_negotiation.cc:407] Connection header received
0927 07:06:19.481104 (+ 3278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.481110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.481207 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.481347 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:19.484058 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.485325 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.488587 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.489053 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.489161 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.495822 (+ 6661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:19.495831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:19.495839 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:19.495893 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:19.498966 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.500399 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.500406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.500409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.500588 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.501307 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.501313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.501319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.501764 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:19.501938 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.503345 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.503562 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:06:19.505867 32284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.476729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.67:45621)
0927 07:06:19.477654 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.477702 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.480736 (+ 3034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.481817 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.481835 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.482792 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:19.483850 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.483878 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.485517 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.485526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.488415 (+ 2889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.488435 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.489328 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.489338 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.489703 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.490797 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:19.490846 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:19.495486 (+ 4640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:19.499152 (+ 3666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.499165 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.499191 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.500153 (+ 962us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.500778 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.500783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.500789 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.501129 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.502034 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:19.502044 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:19.502731 (+ 687us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.503110 (+ 379us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.503887 (+ 777us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":350,"threads_started":1}
I20260927 07:06:19.527076 32260 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:41603
I20260927 07:06:19.527076 32394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:41603 every 8 connection(s)
I20260927 07:06:19.528501 32260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:19.529211 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.513255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45023 (local address 127.30.16.126:39301)
0927 07:06:19.513438 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.513444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.514134 (+ 690us) server_negotiation.cc:407] Connection header received
0927 07:06:19.514344 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.514349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.514435 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.514540 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:19.515853 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.516870 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.519100 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.519702 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.519862 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.522845 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:19.522854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:19.522861 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:19.522914 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:19.526236 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.527085 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.527091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.527094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.527261 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.527753 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.527758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.527762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.528042 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:19.528202 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.528612 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.528848 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:06:19.529352 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.513350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.30.16.67:45023)
0927 07:06:19.514021 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.514040 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.514202 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.514587 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.514592 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.515357 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:19.515696 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.515706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.517071 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.517083 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.518934 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.518954 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.519980 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.519994 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.520244 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.521162 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:19.521193 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:19.522597 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:19.526380 (+ 3783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.526385 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.526389 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.526953 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.527427 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:19.527431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:19.527435 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:19.527633 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:19.528157 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:19.528162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:19.528350 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.528485 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.528848 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":369,"threads_started":1}
I20260927 07:06:19.530947 32395 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39301
I20260927 07:06:19.531267 32395 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:19.532357 32395 heartbeater.cc:511] Master 127.30.16.126:39301 requested a full tablet report, sending...
I20260927 07:06:19.532629 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 32260
I20260927 07:06:19.532840 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:19.533384 30785 external_mini_cluster.cc:1428] Setting key 7ef7ebdfecd240477f5c4ebdc1a3a7b1
I20260927 07:06:19.534266 31909 ts_manager.cc:194] Registered new tserver with Master: f516a174a7de4d60818610e13399d368 (127.30.16.67:41603)
I20260927 07:06:19.535538 31909 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:45621
I20260927 07:06:19.536782 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:06:19 dist-test-slave-8vqf krb5kdc[31865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:06:19 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:19.582523 32005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.568436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.30.16.126:39301)
0927 07:06:19.568686 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.568695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.568992 (+ 297us) server_negotiation.cc:407] Connection header received
0927 07:06:19.569224 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.569229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.569294 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.569411 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:19.570883 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.571641 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.572906 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.573334 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.573476 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.577502 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:19.577507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:19.577510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:19.577546 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:19.580018 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.580593 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.580596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.580599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.580711 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:19.581069 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:19.581074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:19.581078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:19.581362 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:19.581624 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.581919 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.582138 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 07:06:19.586892 31910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:19.593704 31910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:19.629185 32412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.606642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38067 (local address 127.0.0.1:40430)
0927 07:06:19.619761 (+ 13119us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.619782 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.619927 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.620965 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.620972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.621001 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:19.621501 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.621518 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.623746 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.623753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.625894 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.625909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.628119 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.628125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.628323 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.628378 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.628516 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.628706 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12983,"mutex_wait_us":37,"thread_start_us":377,"threads_started":1}
I20260927 07:06:19.629810 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.606663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.30.16.67:41603)
0927 07:06:19.621734 (+ 15071us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.621742 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.621775 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:06:19.621873 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.621883 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.622161 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.622365 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:19.623073 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.624783 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.627331 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.628693 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.628873 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.629106 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.629326 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.629496 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14892,"thread_start_us":14648,"threads_started":1}
I20260927 07:06:19.630070 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.607266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.30.16.65:38067)
0927 07:06:19.620402 (+ 13136us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.620408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.620434 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:19.620520 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.620526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.620726 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.620970 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:19.621737 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.623565 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.626071 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.627627 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.627751 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.627983 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.629602 (+ 1619us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.629779 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13006,"thread_start_us":12904,"threads_started":1}
I20260927 07:06:19.630229 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.608117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33882 (local address 127.30.16.66:32805)
0927 07:06:19.618992 (+ 10875us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.619006 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.622121 (+ 3115us) server_negotiation.cc:407] Connection header received
0927 07:06:19.622311 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.622320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.622616 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.622827 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:19.623717 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.625051 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.627550 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.628895 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.628997 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.629199 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.629778 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.629947 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10620,"thread_start_us":10905,"threads_started":1}
I20260927 07:06:19.629812 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.606355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:41603 (local address 127.0.0.1:33240)
0927 07:06:19.607365 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.607401 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.607566 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.622360 (+ 14794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.622364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.622388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:19.622850 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.622862 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.624921 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.624926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.627176 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.627187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.628944 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.628951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.629123 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.629161 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.629270 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.629412 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"mutex_wait_us":37,"thread_start_us":15378,"threads_started":1}
I20260927 07:06:19.630906 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.607785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:32805 (local address 127.0.0.1:33882)
0927 07:06:19.621986 (+ 14201us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.622002 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.622146 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.623062 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.623066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.623089 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:19.623508 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.623519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.625187 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.625192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.627348 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.627362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.629396 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.629401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.629584 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.629614 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.630397 (+ 783us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.630539 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14086,"mutex_wait_us":30,"thread_start_us":11688,"threads_started":1}
I20260927 07:06:19.634014 32325 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5abaf7b014a60804c4211af569bfe (DEFAULT_TABLE table=test-table [id=f0d514bbc5b7437d9581c638202e9837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.634281 32186 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5abaf7b014a60804c4211af569bfe (DEFAULT_TABLE table=test-table [id=f0d514bbc5b7437d9581c638202e9837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.634457 32047 tablet_service.cc:1467] Processing CreateTablet for tablet 69e5abaf7b014a60804c4211af569bfe (DEFAULT_TABLE table=test-table [id=f0d514bbc5b7437d9581c638202e9837]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.635322 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5abaf7b014a60804c4211af569bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.635655 32047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5abaf7b014a60804c4211af569bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.635673 32186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69e5abaf7b014a60804c4211af569bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.646473 32414 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Bootstrap starting.
I20260927 07:06:19.650434 32414 tablet_bootstrap.cc:654] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.651715 32414 log.cc:824] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:19.653110 32416 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Bootstrap starting.
I20260927 07:06:19.653898 32415 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Bootstrap starting.
I20260927 07:06:19.655102 32414 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: No bootstrap required, opened a new log
I20260927 07:06:19.655411 32414 ts_tablet_manager.cc:1397] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 07:06:19.658690 32416 tablet_bootstrap.cc:654] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.659969 32416 log.cc:824] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:19.660597 32415 tablet_bootstrap.cc:654] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.661738 32415 log.cc:824] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:19.663269 32416 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: No bootstrap required, opened a new log
I20260927 07:06:19.663606 32416 ts_tablet_manager.cc:1397] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260927 07:06:19.665829 32414 raft_consensus.cc:348] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.666308 32414 raft_consensus.cc:374] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.666440 32414 raft_consensus.cc:729] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f516a174a7de4d60818610e13399d368, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.667109 32414 consensus_queue.cc:260] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.668427 32395 heartbeater.cc:503] Master 127.30.16.126:39301 was elected leader, sending a full tablet report...
I20260927 07:06:19.668958 32414 ts_tablet_manager.cc:1428] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 07:06:19.673985 32416 raft_consensus.cc:348] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.674461 32416 raft_consensus.cc:374] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.674640 32416 raft_consensus.cc:729] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.674921 32415 tablet_bootstrap.cc:492] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: No bootstrap required, opened a new log
I20260927 07:06:19.675220 32415 ts_tablet_manager.cc:1397] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.010s
I20260927 07:06:19.675287 32416 consensus_queue.cc:260] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.677057 32416 ts_tablet_manager.cc:1428] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 07:06:19.683928 32415 raft_consensus.cc:348] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.684325 32415 raft_consensus.cc:374] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.684429 32415 raft_consensus.cc:729] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c5964bd6dc423a9a4663d1a30884c9, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.684983 32415 consensus_queue.cc:260] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.685984 32256 heartbeater.cc:503] Master 127.30.16.126:39301 was elected leader, sending a full tablet report...
I20260927 07:06:19.686496 32415 ts_tablet_manager.cc:1428] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Time spent starting tablet: real 0.011s user 0.005s sys 0.006s
W20260927 07:06:19.728138 32118 tablet.cc:2369] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:19.756029 32257 tablet.cc:2369] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:19.786258 32396 tablet.cc:2369] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:06:19.794895 32421 raft_consensus.cc:482] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:19.795239 32421 raft_consensus.cc:504] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.797101 32421 leader_election.cc:288] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f516a174a7de4d60818610e13399d368 (127.30.16.67:41603), 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805)
I20260927 07:06:19.803992 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.797751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:34143 (local address 127.30.16.67:41603)
0927 07:06:19.797932 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.797939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.798604 (+ 665us) server_negotiation.cc:407] Connection header received
0927 07:06:19.798852 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.798858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.798936 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.799050 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:19.799632 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.800413 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.802197 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.803056 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.803224 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.803316 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.803595 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.803718 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 07:06:19.804009 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.797544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60243 (local address 127.30.16.66:32805)
0927 07:06:19.797777 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.797786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.798463 (+ 677us) server_negotiation.cc:407] Connection header received
0927 07:06:19.798666 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.798671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.798758 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.798838 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:19.799563 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.800298 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.802243 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.803071 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.803255 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.803325 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.803636 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.803752 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:06:19.804126 32424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.797563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:41603 (local address 127.30.16.65:34143)
0927 07:06:19.798458 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.798476 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.798652 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.799066 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.799072 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.799100 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:19.799462 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.799470 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.800555 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.800558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.802047 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.802057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.803275 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.803283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.803440 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.803468 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.803543 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.803687 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":36,"thread_start_us":428,"threads_started":1}
I20260927 07:06:19.804633 32423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.797501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:32805 (local address 127.30.16.65:60243)
0927 07:06:19.798345 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:19.798360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:19.798488 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:19.798893 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:19.798898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:19.798923 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:19.799408 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.799420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.800435 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.800441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.802061 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:19.802071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.803328 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.803337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:19.803464 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.803486 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:06:19.803595 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:06:19.803717 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":26,"spinlock_wait_cycles":27392,"thread_start_us":526,"threads_started":1}
I20260927 07:06:19.805255 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5abaf7b014a60804c4211af569bfe" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368" is_pre_election: true
I20260927 07:06:19.805824 32345 raft_consensus.cc:2454] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:19.805812 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5abaf7b014a60804c4211af569bfe" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9" is_pre_election: true
I20260927 07:06:19.806360 32206 raft_consensus.cc:2454] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:19.806615 32000 leader_election.cc:302] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, f516a174a7de4d60818610e13399d368; no voters:
I20260927 07:06:19.807070 32421 raft_consensus.cc:2781] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:19.807197 32421 raft_consensus.cc:482] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:19.807317 32421 raft_consensus.cc:3037] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:19.809540 32421 raft_consensus.cc:504] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.810184 32421 leader_election.cc:288] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Requested vote from peers f516a174a7de4d60818610e13399d368 (127.30.16.67:41603), 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805)
I20260927 07:06:19.810781 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5abaf7b014a60804c4211af569bfe" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368"
I20260927 07:06:19.810861 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69e5abaf7b014a60804c4211af569bfe" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9"
I20260927 07:06:19.811017 32345 raft_consensus.cc:3037] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:19.811044 32206 raft_consensus.cc:3037] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:19.814208 32206 raft_consensus.cc:2454] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:19.814208 32345 raft_consensus.cc:2454] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:19.814678 32000 leader_election.cc:302] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, f516a174a7de4d60818610e13399d368; no voters:
I20260927 07:06:19.814939 32421 raft_consensus.cc:2781] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:19.815374 32421 raft_consensus.cc:686] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Running, Role: LEADER
I20260927 07:06:19.815869 32421 consensus_queue.cc:237] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } }
I20260927 07:06:19.820425 31910 catalog_manager.cc:5495] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e118767c524f15aa5932da4a840bf2 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:19.888518 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:19.883104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40440 (local address 127.30.16.65:38067)
0927 07:06:19.883299 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:19.883303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:19.883495 (+ 192us) server_negotiation.cc:407] Connection header received
0927 07:06:19.883758 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:19.883762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:19.883838 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:19.883959 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:19.884260 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.884954 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.886359 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:19.886827 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:19.886924 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:19.887490 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:19.888009 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:19.888116 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:19.888236 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 07:06:19.888300 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:06:19.896978 32206 raft_consensus.cc:1261] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:19.896970 32345 raft_consensus.cc:1261] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:19.897945 32425 consensus_queue.cc:1035] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:19.898306 32421 consensus_queue.cc:1035] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:19.910123 32428 mvcc.cc:205] Tried to move back new op lower bound from 7333858426448121856 to 7333858426129653760. Current Snapshot: MvccSnapshot[applied={T|T < 7333858426448121856}]
I20260927 07:06:19.929687 31909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:19.929993 31909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:19.934115 31909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Sending DeleteTablet for 3 replicas of tablet 69e5abaf7b014a60804c4211af569bfe
I20260927 07:06:19.935367 32325 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5abaf7b014a60804c4211af569bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:19 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33240
I20260927 07:06:19.935581 32047 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5abaf7b014a60804c4211af569bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:19 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:40430
I20260927 07:06:19.936326 32186 tablet_service.cc:1514] Processing DeleteTablet for tablet 69e5abaf7b014a60804c4211af569bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:19 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33882
I20260927 07:06:19.937661 31909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:19.938882 31909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:19.942905 32439 tablet_replica.cc:330] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: stopping tablet replica
I20260927 07:06:19.943665 32440 tablet_replica.cc:330] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: stopping tablet replica
I20260927 07:06:19.943840 32438 tablet_replica.cc:330] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: stopping tablet replica
I20260927 07:06:19.944331 32438 raft_consensus.cc:2229] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:19.944883 32438 raft_consensus.cc:2258] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:19.944319 32440 raft_consensus.cc:2229] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:19.944891 32439 raft_consensus.cc:2229] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:19.946280 32440 raft_consensus.cc:2258] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:19.947449 32438 ts_tablet_manager.cc:1905] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:19.948156 32047 tablet_service.cc:1467] Processing CreateTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (DEFAULT_TABLE table=test-table [id=e23047914b944e28bf169f85a5f670a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.948143 32186 tablet_service.cc:1467] Processing CreateTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (DEFAULT_TABLE table=test-table [id=e23047914b944e28bf169f85a5f670a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.948627 32047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa3572e8a5a4b4eaf293bd99aa4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.948639 32186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa3572e8a5a4b4eaf293bd99aa4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.948163 32439 raft_consensus.cc:2258] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:19.952514 32439 ts_tablet_manager.cc:1905] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:19.952514 32440 ts_tablet_manager.cc:1905] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:19.955904 32416 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Bootstrap starting.
I20260927 07:06:19.956087 32415 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Bootstrap starting.
I20260927 07:06:19.956934 32325 tablet_service.cc:1467] Processing CreateTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (DEFAULT_TABLE table=test-table [id=e23047914b944e28bf169f85a5f670a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:19.957365 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa3572e8a5a4b4eaf293bd99aa4f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:19.957835 32415 tablet_bootstrap.cc:654] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.957844 32416 tablet_bootstrap.cc:654] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.962857 32414 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Bootstrap starting.
I20260927 07:06:19.964169 32414 tablet_bootstrap.cc:654] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:19.968039 32416 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: No bootstrap required, opened a new log
I20260927 07:06:19.968220 32416 ts_tablet_manager.cc:1397] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20260927 07:06:19.968384 32415 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: No bootstrap required, opened a new log
I20260927 07:06:19.968547 32415 ts_tablet_manager.cc:1397] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260927 07:06:19.969199 32416 raft_consensus.cc:348] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.969473 32416 raft_consensus.cc:374] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.969362 32415 raft_consensus.cc:348] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.969595 32415 raft_consensus.cc:374] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.969606 32416 raft_consensus.cc:729] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.969695 32415 raft_consensus.cc:729] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c5964bd6dc423a9a4663d1a30884c9, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.969950 32415 consensus_queue.cc:260] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.969973 32416 consensus_queue.cc:260] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.970585 32415 ts_tablet_manager.cc:1428] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 07:06:19.974999 32438 ts_tablet_manager.cc:1918] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:19.975207 32438 log.cc:1196] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/69e5abaf7b014a60804c4211af569bfe
I20260927 07:06:19.975827 32438 ts_tablet_manager.cc:1939] T 69e5abaf7b014a60804c4211af569bfe P f516a174a7de4d60818610e13399d368: Deleting consensus metadata
I20260927 07:06:19.976804 32414 tablet_bootstrap.cc:492] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: No bootstrap required, opened a new log
I20260927 07:06:19.976961 32414 ts_tablet_manager.cc:1397] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 07:06:19.977802 32414 raft_consensus.cc:348] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.978031 32414 raft_consensus.cc:374] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:19.977977 31897 catalog_manager.cc:4841] TS f516a174a7de4d60818610e13399d368 (127.30.16.67:41603): tablet 69e5abaf7b014a60804c4211af569bfe (table test-table [id=f0d514bbc5b7437d9581c638202e9837]) successfully deleted
I20260927 07:06:19.978129 32414 raft_consensus.cc:729] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f516a174a7de4d60818610e13399d368, State: Initialized, Role: FOLLOWER
I20260927 07:06:19.978417 32414 consensus_queue.cc:260] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:19.979133 32416 ts_tablet_manager.cc:1428] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20260927 07:06:19.979215 32439 ts_tablet_manager.cc:1918] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:19.979403 32439 log.cc:1196] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/69e5abaf7b014a60804c4211af569bfe
I20260927 07:06:19.980037 32439 ts_tablet_manager.cc:1939] T 69e5abaf7b014a60804c4211af569bfe P 01e118767c524f15aa5932da4a840bf2: Deleting consensus metadata
I20260927 07:06:19.980118 32414 ts_tablet_manager.cc:1428] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:19.981971 31894 catalog_manager.cc:4841] TS 01e118767c524f15aa5932da4a840bf2 (127.30.16.65:38067): tablet 69e5abaf7b014a60804c4211af569bfe (table test-table [id=f0d514bbc5b7437d9581c638202e9837]) successfully deleted
I20260927 07:06:19.985065 32440 ts_tablet_manager.cc:1918] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:19.985241 32440 log.cc:1196] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/69e5abaf7b014a60804c4211af569bfe
I20260927 07:06:19.985823 32440 ts_tablet_manager.cc:1939] T 69e5abaf7b014a60804c4211af569bfe P 87c5964bd6dc423a9a4663d1a30884c9: Deleting consensus metadata
I20260927 07:06:19.987663 31894 catalog_manager.cc:4841] TS 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805): tablet 69e5abaf7b014a60804c4211af569bfe (table test-table [id=f0d514bbc5b7437d9581c638202e9837]) successfully deleted
I20260927 07:06:20.229034 32433 raft_consensus.cc:482] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:20.229328 32433 raft_consensus.cc:504] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.230301 32433 leader_election.cc:288] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805), f516a174a7de4d60818610e13399d368 (127.30.16.67:41603)
I20260927 07:06:20.230963 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa3572e8a5a4b4eaf293bd99aa4f533" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368" is_pre_election: true
I20260927 07:06:20.230940 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa3572e8a5a4b4eaf293bd99aa4f533" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9" is_pre_election: true
I20260927 07:06:20.231228 32206 raft_consensus.cc:2454] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:20.231228 32345 raft_consensus.cc:2454] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:20.231722 32000 leader_election.cc:302] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, f516a174a7de4d60818610e13399d368; no voters:
I20260927 07:06:20.232036 32433 raft_consensus.cc:2781] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:20.232188 32433 raft_consensus.cc:482] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:20.232296 32433 raft_consensus.cc:3037] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.235759 32433 raft_consensus.cc:504] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.236505 32433 leader_election.cc:288] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805), f516a174a7de4d60818610e13399d368 (127.30.16.67:41603)
I20260927 07:06:20.237088 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa3572e8a5a4b4eaf293bd99aa4f533" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9"
I20260927 07:06:20.237080 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa3572e8a5a4b4eaf293bd99aa4f533" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368"
I20260927 07:06:20.237321 32206 raft_consensus.cc:3037] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.237326 32345 raft_consensus.cc:3037] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.239614 32206 raft_consensus.cc:2454] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:20.240167 31997 leader_election.cc:302] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, 87c5964bd6dc423a9a4663d1a30884c9; no voters:
I20260927 07:06:20.240314 32345 raft_consensus.cc:2454] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:20.240597 32433 raft_consensus.cc:2781] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:20.240846 32433 raft_consensus.cc:686] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Running, Role: LEADER
I20260927 07:06:20.241232 32433 consensus_queue.cc:237] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.244298 31910 catalog_manager.cc:5495] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e118767c524f15aa5932da4a840bf2 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } health_report { overall_health: UNKNOWN } } }
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:20.308570 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.285134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39301 (local address 127.0.0.1:41940)
0927 07:06:20.285818 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:20.285841 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:20.286017 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:20.286413 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:20.286418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:20.287106 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:20.287295 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.287304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.288496 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.288504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.289426 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.289439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.290108 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.290114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.290303 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.299111 (+ 8808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:20.299135 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:20.301644 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:20.305758 (+ 4114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:20.305774 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:20.305783 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:20.306300 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:20.306733 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:20.306739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:20.306742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:20.306875 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:20.307409 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:20.307417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:20.307598 (+ 181us) client_negotiation.cc:769] Sending connection context
0927 07:06:20.307769 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 07:06:20.308086 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
I20260927 07:06:20.308591 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.285205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.30.16.126:39301)
0927 07:06:20.286163 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.286167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.286183 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:06:20.286223 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.286226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.286277 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.286389 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.287466 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.288351 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.289566 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.289952 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.290088 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.301891 (+ 11803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.301897 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.301900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.301931 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.305582 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.306426 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.306434 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.306437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.306619 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.306989 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.306994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.306997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.307272 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.307418 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.307908 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.308209 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":767,"thread_start_us":615,"threads_started":1}
I20260927 07:06:20.314859 31910 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:41940:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:06:20.315948 31910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:20.323189 32047 tablet_service.cc:1467] Processing CreateTablet for tablet 3170c20aef134f51a662720d0a323e7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06a3b7e6282497fa556ec1547c9690f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:20.323433 32186 tablet_service.cc:1467] Processing CreateTablet for tablet 3170c20aef134f51a662720d0a323e7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06a3b7e6282497fa556ec1547c9690f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:20.323824 32047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170c20aef134f51a662720d0a323e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:20.324040 32186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170c20aef134f51a662720d0a323e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:20.329564 32415 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: Bootstrap starting.
I20260927 07:06:20.330585 32325 tablet_service.cc:1467] Processing CreateTablet for tablet 3170c20aef134f51a662720d0a323e7a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e06a3b7e6282497fa556ec1547c9690f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:20.331105 32325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170c20aef134f51a662720d0a323e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:20.331454 32415 tablet_bootstrap.cc:654] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:20.333398 32416 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: Bootstrap starting.
I20260927 07:06:20.335843 32416 tablet_bootstrap.cc:654] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:20.338030 32414 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: Bootstrap starting.
I20260927 07:06:20.339636 32414 tablet_bootstrap.cc:654] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:20.348378 32415 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: No bootstrap required, opened a new log
I20260927 07:06:20.348663 32415 ts_tablet_manager.cc:1397] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.015s
I20260927 07:06:20.348685 32416 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: No bootstrap required, opened a new log
I20260927 07:06:20.348856 32416 ts_tablet_manager.cc:1397] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260927 07:06:20.349633 32415 raft_consensus.cc:348] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.349643 32416 raft_consensus.cc:348] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.349918 32416 raft_consensus.cc:374] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:20.350014 32416 raft_consensus.cc:729] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Initialized, Role: FOLLOWER
I20260927 07:06:20.350055 32414 tablet_bootstrap.cc:492] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: No bootstrap required, opened a new log
I20260927 07:06:20.350044 32415 raft_consensus.cc:374] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:20.350229 32414 ts_tablet_manager.cc:1397] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260927 07:06:20.350231 32415 raft_consensus.cc:729] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c5964bd6dc423a9a4663d1a30884c9, State: Initialized, Role: FOLLOWER
I20260927 07:06:20.350309 32416 consensus_queue.cc:260] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.350742 32415 consensus_queue.cc:260] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.351238 32433 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.351255 32422 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.351552 32422 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.351585 32433 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.351035 32414 raft_consensus.cc:348] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.352358 32414 raft_consensus.cc:374] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:20.352443 32414 raft_consensus.cc:729] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f516a174a7de4d60818610e13399d368, State: Initialized, Role: FOLLOWER
I20260927 07:06:20.352715 32415 ts_tablet_manager.cc:1428] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 07:06:20.352686 32414 consensus_queue.cc:260] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.353119 32420 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.353406 32420 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.353763 32414 ts_tablet_manager.cc:1428] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:06:20.354389 32416 ts_tablet_manager.cc:1428] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 07:06:20.419149 32433 raft_consensus.cc:482] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:20.419413 32433 raft_consensus.cc:504] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.420081 32433 leader_election.cc:288] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805), f516a174a7de4d60818610e13399d368 (127.30.16.67:41603)
I20260927 07:06:20.420593 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170c20aef134f51a662720d0a323e7a" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9" is_pre_election: true
I20260927 07:06:20.420655 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170c20aef134f51a662720d0a323e7a" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368" is_pre_election: true
I20260927 07:06:20.420883 32345 raft_consensus.cc:2454] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:20.420884 32206 raft_consensus.cc:2454] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 0.
I20260927 07:06:20.421315 32000 leader_election.cc:302] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, 87c5964bd6dc423a9a4663d1a30884c9, f516a174a7de4d60818610e13399d368; no voters:
I20260927 07:06:20.421612 32433 raft_consensus.cc:2781] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:20.421738 32433 raft_consensus.cc:482] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:20.421851 32433 raft_consensus.cc:3037] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.423931 32433 raft_consensus.cc:504] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.424501 32433 leader_election.cc:288] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Requested vote from peers 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805), f516a174a7de4d60818610e13399d368 (127.30.16.67:41603)
I20260927 07:06:20.424925 32206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170c20aef134f51a662720d0a323e7a" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c5964bd6dc423a9a4663d1a30884c9"
I20260927 07:06:20.425005 32345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170c20aef134f51a662720d0a323e7a" candidate_uuid: "01e118767c524f15aa5932da4a840bf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f516a174a7de4d60818610e13399d368"
I20260927 07:06:20.425177 32206 raft_consensus.cc:3037] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.425212 32345 raft_consensus.cc:3037] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:20.428100 32206 raft_consensus.cc:2454] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:20.428100 32345 raft_consensus.cc:2454] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01e118767c524f15aa5932da4a840bf2 in term 1.
I20260927 07:06:20.428529 31997 leader_election.cc:302] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01e118767c524f15aa5932da4a840bf2, 87c5964bd6dc423a9a4663d1a30884c9; no voters:
I20260927 07:06:20.428855 32433 raft_consensus.cc:2781] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:20.429049 32433 raft_consensus.cc:686] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 01e118767c524f15aa5932da4a840bf2, State: Running, Role: LEADER
I20260927 07:06:20.429288 32433 consensus_queue.cc:237] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } }
I20260927 07:06:20.430812 32425 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: TxnStatusTablet state changed. Reason: New leader 01e118767c524f15aa5932da4a840bf2. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.431054 32425 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:20.431612 32456 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:06:20.431823 32456 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:06:20.432301 31910 catalog_manager.cc:5495] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 01e118767c524f15aa5932da4a840bf2 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:20.454223 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.447813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.30.16.65:38067)
0927 07:06:20.448478 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.448484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.448504 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:06:20.448573 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.448578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.448673 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.448783 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:20.449175 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.450307 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.452316 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.452819 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.452944 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.453490 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:20.453598 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:20.453677 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.453873 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.453944 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":392,"threads_started":1}
I20260927 07:06:20.454321 32450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.447857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38067 (local address 127.0.0.1:40456)
0927 07:06:20.448162 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:20.448190 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:20.448343 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:20.448802 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:20.448805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:20.448815 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:20.448992 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.448999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.450441 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.450447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.452145 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.452160 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.452993 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.452999 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.453206 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.453281 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:20.453703 (+ 422us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:20.453725 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:06:20.453817 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 07:06:20.453997 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 07:06:20.458338 32206 raft_consensus.cc:1261] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:20.458392 32345 raft_consensus.cc:1261] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:20.458993 32433 consensus_queue.cc:1035] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:20.459290 32425 consensus_queue.cc:1035] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:20.463837 32420 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: TxnStatusTablet state changed. Reason: New leader 01e118767c524f15aa5932da4a840bf2. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.464107 32420 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.465469 32422 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: TxnStatusTablet state changed. Reason: New leader 01e118767c524f15aa5932da4a840bf2. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.465755 32422 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.466490 32461 mvcc.cc:205] Tried to move back new op lower bound from 7333858428751761408 to 7333858428640571392. Current Snapshot: MvccSnapshot[applied={T|T < 7333858428751761408}]
I20260927 07:06:20.467293 32420 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.467522 32420 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P f516a174a7de4d60818610e13399d368: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.471001 32425 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.471261 32425 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:20.471532 32425 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.471747 32425 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 01e118767c524f15aa5932da4a840bf2: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:20.480979 32422 tablet_replica.cc:439] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "01e118767c524f15aa5932da4a840bf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01e118767c524f15aa5932da4a840bf2" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38067 } } peers { permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 } } peers { permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 } } }
I20260927 07:06:20.481268 32422 tablet_replica.cc:442] T 3170c20aef134f51a662720d0a323e7a P 87c5964bd6dc423a9a4663d1a30884c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:20.500874 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.494971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:37711 (local address 127.30.16.65:38067)
0927 07:06:20.495136 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.495142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.495786 (+ 644us) server_negotiation.cc:407] Connection header received
0927 07:06:20.496045 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.496050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.496125 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.496212 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:20.496738 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.497714 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.499163 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.499599 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.499743 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.500159 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:20.500271 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:20.500372 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.500494 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.500560 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:06:20.501220 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.494971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38067 (local address 127.30.16.65:37711)
0927 07:06:20.495692 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:20.495709 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:20.495870 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:20.496232 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:20.496236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:20.496246 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:20.496583 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.496592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.497834 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.497842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.499036 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:20.499048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.499790 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.499798 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:20.499951 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.500003 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:20.500368 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:20.500377 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 07:06:20.500477 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:06:20.500605 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260927 07:06:20.512545 32345 raft_consensus.cc:1261] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:20.513073 32433 consensus_queue.cc:1035] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f516a174a7de4d60818610e13399d368" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:20.513561 32205 raft_consensus.cc:1261] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Refusing update from remote peer 01e118767c524f15aa5932da4a840bf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:20.514106 32433 consensus_queue.cc:1035] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c5964bd6dc423a9a4663d1a30884c9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:20.523594 32460 mvcc.cc:205] Tried to move back new op lower bound from 7333858428974313472 to 7333858427870822400. Current Snapshot: MvccSnapshot[applied={T|T < 7333858428974313472}]
I20260927 07:06:20.595669 31909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:20.595942 31909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:20.599174 31909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f71d67987d9f4a93b416fd4c4f80d625: Sending DeleteTablet for 3 replicas of tablet dfa3572e8a5a4b4eaf293bd99aa4f533
I20260927 07:06:20.600106 32186 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33882
I20260927 07:06:20.600128 32325 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33240
I20260927 07:06:20.600543 32047 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa3572e8a5a4b4eaf293bd99aa4f533 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:40430
I20260927 07:06:20.600849 32476 tablet_replica.cc:330] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: stopping tablet replica
I20260927 07:06:20.601095 32476 raft_consensus.cc:2229] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:20.601401 32476 raft_consensus.cc:2258] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:20.602771 32476 ts_tablet_manager.cc:1905] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:20.605144 32477 tablet_replica.cc:330] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: stopping tablet replica
I20260927 07:06:20.605484 32477 raft_consensus.cc:2229] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:20.605939 32477 raft_consensus.cc:2258] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:20.613577 32478 tablet_replica.cc:330] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: stopping tablet replica
I20260927 07:06:20.613811 32477 ts_tablet_manager.cc:1905] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:20.613822 32478 raft_consensus.cc:2229] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:20.614287 32478 raft_consensus.cc:2258] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:20.615499 32478 ts_tablet_manager.cc:1905] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:20.622237 32477 ts_tablet_manager.cc:1918] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:20.622640 32477 log.cc:1196] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/dfa3572e8a5a4b4eaf293bd99aa4f533
I20260927 07:06:20.623107 32478 ts_tablet_manager.cc:1918] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:20.623245 32478 log.cc:1196] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/dfa3572e8a5a4b4eaf293bd99aa4f533
I20260927 07:06:20.623718 32478 ts_tablet_manager.cc:1939] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 01e118767c524f15aa5932da4a840bf2: Deleting consensus metadata
I20260927 07:06:20.623952 32476 ts_tablet_manager.cc:1918] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:20.624109 32477 ts_tablet_manager.cc:1939] T dfa3572e8a5a4b4eaf293bd99aa4f533 P 87c5964bd6dc423a9a4663d1a30884c9: Deleting consensus metadata
I20260927 07:06:20.624933 31894 catalog_manager.cc:4841] TS 01e118767c524f15aa5932da4a840bf2 (127.30.16.65:38067): tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (table test-table [id=e23047914b944e28bf169f85a5f670a7]) successfully deleted
I20260927 07:06:20.625181 32476 log.cc:1196] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/dfa3572e8a5a4b4eaf293bd99aa4f533
I20260927 07:06:20.625730 32476 ts_tablet_manager.cc:1939] T dfa3572e8a5a4b4eaf293bd99aa4f533 P f516a174a7de4d60818610e13399d368: Deleting consensus metadata
I20260927 07:06:20.626853 31897 catalog_manager.cc:4841] TS f516a174a7de4d60818610e13399d368 (127.30.16.67:41603): tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (table test-table [id=e23047914b944e28bf169f85a5f670a7]) successfully deleted
I20260927 07:06:20.626853 31894 catalog_manager.cc:4841] TS 87c5964bd6dc423a9a4663d1a30884c9 (127.30.16.66:32805): tablet dfa3572e8a5a4b4eaf293bd99aa4f533 (table test-table [id=e23047914b944e28bf169f85a5f670a7]) successfully deleted
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:20.636046 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.612668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40460 (local address 127.30.16.65:38067)
0927 07:06:20.612841 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.612849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.618104 (+ 5255us) server_negotiation.cc:407] Connection header received
0927 07:06:20.618605 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.618613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.618734 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.618826 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.620825 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.622024 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.623963 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.624655 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.624760 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.629964 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.629979 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.630015 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.630053 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.633112 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.633836 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.633844 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.633847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.634001 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.634403 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.634408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.634410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.634835 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.635160 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.635483 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.635709 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 07:06:20.636885 32112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40460
I20260927 07:06:20.654321 32451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.640074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.30.16.126:39301)
0927 07:06:20.640338 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.640348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.640780 (+ 432us) server_negotiation.cc:407] Connection header received
0927 07:06:20.640989 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.640994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.641077 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.641182 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.642404 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.643592 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.644766 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.645449 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.645579 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.648045 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.648053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.648058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.648101 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.651656 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.652338 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.652344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.652348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.652497 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.652831 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.652837 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.652843 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.653209 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.653447 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.653722 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.653975 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260927 07:06:20.654968 31909 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41954
I20260927 07:06:20.656922 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31981
I20260927 07:06:20.672394 32458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.657845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.30.16.65:38067)
0927 07:06:20.658074 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.658083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.658543 (+ 460us) server_negotiation.cc:407] Connection header received
0927 07:06:20.658867 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.658871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.658939 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.659061 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.660432 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.661622 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.662870 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.663578 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.663707 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.665906 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.665913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.665918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.665959 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.669399 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.670175 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.670186 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.670189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.670376 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.670772 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.670777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.670781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.671136 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.671406 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.671631 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.671976 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 07:06:20.673156 32112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40466
W20260927 07:06:20.673756 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31981: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:20.673969 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31981
I20260927 07:06:20.696832 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32121
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:20.712141 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.697620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33886 (local address 127.30.16.66:32805)
0927 07:06:20.698107 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.698112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.698128 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:06:20.698183 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.698187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.698264 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.698353 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.699516 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.700308 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.701497 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.701902 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.702000 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.706523 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.706539 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.706619 (+ 80us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.706663 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.709719 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.710373 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.710380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.710383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.710521 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.710888 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.710891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.710893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.711226 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.711449 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.711707 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.711880 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":226,"threads_started":1}
W20260927 07:06:20.712828 32251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33886
W20260927 07:06:20.713460 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:20.713604 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 32121
I20260927 07:06:20.732929 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32260
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[31865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492779, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:20.748055 32485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:20.733725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.30.16.67:41603)
0927 07:06:20.734275 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:20.734280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:20.734297 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:06:20.734355 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:20.734358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:20.734434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:20.734524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:20.735798 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.736588 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.737771 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:20.738207 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:20.738302 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:20.742491 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:20.742506 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:20.742523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:20.742669 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:20.745539 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.746202 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.746210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.746212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.746384 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:20.746778 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:20.746783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:20.746786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:20.747156 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:20.747396 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:20.747600 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 07:06:20.747769 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
W20260927 07:06:20.748761 32390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33248
W20260927 07:06:20.749481 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32260: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:20.749625 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 32260
I20260927 07:06:20.766600 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 31878
I20260927 07:06:20.767783 31950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:20.889385 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 31878
2026-09-27T07:06:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4749 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[32489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[32489](info): set up 2 sockets
Sep 27 07:06:20 dist-test-slave-8vqf krb5kdc[32489](info): commencing operation
krb5kdc: starting...
W20260927 07:06:21.986857 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:22 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:22.182999 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33611
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:22.512367 32502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:22.512688 32502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:22.512866 32502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:22.520287 32502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:22.520404 32502 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:06:22.520489 32502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:22.520599 32502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:22.520679 32502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:22.520772 32502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:22.531033 32502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:22.532572 32502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:22.534157 32502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:22.550160 32507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:22.550276 32510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:22.550297 32508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:22.551860 32502 server_base.cc:1034] running on GCE node
I20260927 07:06:22.552695 32502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:22.554178 32502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:22.555430 32502 hybrid_clock.cc:648] HybridClock initialized: now 1790492782555356 us; error 56 us; skew 500 ppm
I20260927 07:06:22.555779 32502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:22 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:22.558928 32502 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:22.560388 32502 webserver.cc:460] Webserver started at http://127.30.16.126:38457/ using document root <none> and password file <none>
I20260927 07:06:22.561102 32502 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:22.561241 32502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:22.561556 32502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:22.564575 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "929bf5157abb4b5c921b4638bc16a2f0"
format_stamp: "Formatted at 2026-09-27 07:06:22 on dist-test-slave-8vqf"
server_key: "bf7a40e7aaeff169c9b453182d25cf3a"
server_key_iv: "d4f1b693846f96f4ca3d6de5729645e8"
server_key_version: "encryptionkey@0"
I20260927 07:06:22.565344 32502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "929bf5157abb4b5c921b4638bc16a2f0"
format_stamp: "Formatted at 2026-09-27 07:06:22 on dist-test-slave-8vqf"
server_key: "bf7a40e7aaeff169c9b453182d25cf3a"
server_key_iv: "d4f1b693846f96f4ca3d6de5729645e8"
server_key_version: "encryptionkey@0"
I20260927 07:06:22.570503 32502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:22.573866 32517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:22.575417 32502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:22.575613 32502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "929bf5157abb4b5c921b4638bc16a2f0"
format_stamp: "Formatted at 2026-09-27 07:06:22 on dist-test-slave-8vqf"
server_key: "bf7a40e7aaeff169c9b453182d25cf3a"
server_key_iv: "d4f1b693846f96f4ca3d6de5729645e8"
server_key_version: "encryptionkey@0"
I20260927 07:06:22.575836 32502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:22.591814 32502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:22.595880 32502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:22.596153 32502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:22.617923 32502 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33611
I20260927 07:06:22.617971 32578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33611 every 8 connection(s)
I20260927 07:06:22.619359 32502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:22.621315 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 32502
I20260927 07:06:22.621526 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:22.622337 30785 external_mini_cluster.cc:1428] Setting key 95506acd80c5db43e39e7932070fe510
I20260927 07:06:22.624362 32579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:06:22 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:22.639191 32579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Bootstrap starting.
I20260927 07:06:22.642956 32579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:22.644249 32579 log.cc:824] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:22.647261 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:22.624647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37406 (local address 127.30.16.126:33611)
0927 07:06:22.625429 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:22.625441 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:22.625483 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:06:22.627129 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:22.627150 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:22.627543 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:22.627883 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:22.629751 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:22.631589 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:22.633429 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:22.634098 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:22.634238 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:22.639360 (+ 5122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:22.639372 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:22.639397 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:22.639450 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:22.642733 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:22.643562 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:22.643568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:22.643575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:22.643733 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:22.644170 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:22.644174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:22.644177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:22.644920 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:22.645275 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:22.645668 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 07:06:22.646142 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":358,"threads_started":1}
I20260927 07:06:22.647518 32579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: No bootstrap required, opened a new log
I20260927 07:06:22.654966 32579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } }
I20260927 07:06:22.655330 32579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:22.655458 32579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 929bf5157abb4b5c921b4638bc16a2f0, State: Initialized, Role: FOLLOWER
I20260927 07:06:22.656104 32579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } }
I20260927 07:06:22.656324 32579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:22.656451 32579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:22.656630 32579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:22.658999 32579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } }
I20260927 07:06:22.659507 32579 leader_election.cc:302] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 929bf5157abb4b5c921b4638bc16a2f0; no voters:
I20260927 07:06:22.660104 32579 leader_election.cc:288] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:22.660259 32584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:22.661055 32584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [term 1 LEADER]: Becoming Leader. State: Replica: 929bf5157abb4b5c921b4638bc16a2f0, State: Running, Role: LEADER
I20260927 07:06:22.661572 32584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } }
I20260927 07:06:22.662321 32579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:22.664592 32586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 929bf5157abb4b5c921b4638bc16a2f0. Latest consensus state: current_term: 1 leader_uuid: "929bf5157abb4b5c921b4638bc16a2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } } }
I20260927 07:06:22.664569 32585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "929bf5157abb4b5c921b4638bc16a2f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "929bf5157abb4b5c921b4638bc16a2f0" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33611 } } }
I20260927 07:06:22.665056 32586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:22.665066 32585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:22.666198 32591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:22.672559 32591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:22.682591 32591 catalog_manager.cc:1347] Generated new cluster ID: b5d3a99141a34475b731b7f211fd4305
I20260927 07:06:22.682724 32591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:22.705356 32591 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:22.706827 32591 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:22.723591 32591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Generated new TSK 0
I20260927 07:06:22.724537 32591 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:22.786765 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33611
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:23.115912 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:23.116255 32605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:23.116444 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:23.123838 32605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:23.124004 32605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:23.124161 32605 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:06:23.124257 32605 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:23.124400 32605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:23.134508 32605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:23.135914 32605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:23.137252 32605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:23.152191 32611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:23.152168 32613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:23.152632 32610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:23.153079 32605 server_base.cc:1034] running on GCE node
I20260927 07:06:23.153721 32605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:23.154810 32605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:23.156050 32605 hybrid_clock.cc:648] HybridClock initialized: now 1790492783156004 us; error 67 us; skew 500 ppm
I20260927 07:06:23.156430 32605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:23 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:23.159706 32605 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:23.161237 32605 webserver.cc:460] Webserver started at http://127.30.16.65:37149/ using document root <none> and password file <none>
I20260927 07:06:23.161984 32605 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:23.162168 32605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:23.162508 32605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:23.165669 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "bb54f3198c084ad29daaa89edbfcb954"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "d1bc999dfbeff5df8c89263a278fd0dc"
server_key_iv: "5eaf1f82c3d23e86fdd9dae6fe22cd1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.166464 32605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "bb54f3198c084ad29daaa89edbfcb954"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "d1bc999dfbeff5df8c89263a278fd0dc"
server_key_iv: "5eaf1f82c3d23e86fdd9dae6fe22cd1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.172132 32605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 07:06:23.175834 32620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:23.177372 32605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:06:23.177558 32605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "bb54f3198c084ad29daaa89edbfcb954"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "d1bc999dfbeff5df8c89263a278fd0dc"
server_key_iv: "5eaf1f82c3d23e86fdd9dae6fe22cd1d"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.177774 32605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:23.195539 32605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:23.199658 32605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:23.199967 32605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:23.202631 32605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:23.202769 32605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:23.202924 32605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:23.203022 32605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:06:23 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:23.227793 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.203476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:54907 (local address 127.30.16.126:33611)
0927 07:06:23.203987 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:23.203994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:23.205680 (+ 1686us) server_negotiation.cc:407] Connection header received
0927 07:06:23.208543 (+ 2863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:23.208553 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:23.208671 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:23.208805 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:23.211262 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.211999 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.213626 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.214076 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.214216 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.219361 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:23.219371 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:23.219379 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:23.219435 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:23.223651 (+ 4216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.224508 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.224517 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.224523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.224681 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.225317 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.225322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.225326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.225746 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:23.225893 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:23.227193 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0927 07:06:23.227426 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260927 07:06:23.228891 32630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.204610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.65:54907)
0927 07:06:23.205506 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:23.205557 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:23.208090 (+ 2533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:23.209138 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:23.209152 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:23.210008 (+ 856us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:23.211069 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.211091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.212172 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.212177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.213476 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.213488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.214321 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.214330 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.214654 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.215580 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:23.215617 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:23.219063 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:23.223911 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.223920 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.223943 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.224368 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.224879 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.224887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.224893 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.225176 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.225881 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:23.225888 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:23.226473 (+ 585us) client_negotiation.cc:769] Sending connection context
0927 07:06:23.227107 (+ 634us) client_negotiation.cc:240] Negotiation successful
0927 07:06:23.227672 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":389,"threads_started":1}
I20260927 07:06:23.249696 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.235023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:41877 (local address 127.30.16.126:33611)
0927 07:06:23.235184 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:23.235188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:23.235897 (+ 709us) server_negotiation.cc:407] Connection header received
0927 07:06:23.236099 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:23.236104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:23.236164 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:23.236261 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:23.237189 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.237882 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.239424 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.240324 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.240499 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.243159 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:23.243166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:23.243173 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:23.243216 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:23.246543 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.247516 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.247525 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.247530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.247725 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.248220 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.248226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.248229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.248541 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:23.248697 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:23.249097 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 07:06:23.249328 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:06:23.249797 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.235184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.65:41877)
0927 07:06:23.235792 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:23.235810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:23.235968 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:23.236283 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:23.236287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:23.236761 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:23.237049 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.237057 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.238017 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.238025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.239248 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.239273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.240520 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.240524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.240666 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.241412 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:23.241441 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:23.242944 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:23.246742 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.246747 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.246751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.247384 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.247868 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.247875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.247881 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.248081 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.248666 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:23.248671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:23.248858 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 07:06:23.249009 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 07:06:23.249336 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":348,"threads_started":1}
I20260927 07:06:23.269079 32605 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36561
I20260927 07:06:23.269125 32740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36561 every 8 connection(s)
I20260927 07:06:23.270385 32605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:23.272908 32741 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33611
I20260927 07:06:23.273171 32741 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:23.273854 32741 heartbeater.cc:511] Master 127.30.16.126:33611 requested a full tablet report, sending...
I20260927 07:06:23.276150 32534 ts_manager.cc:194] Registered new tserver with Master: bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561)
I20260927 07:06:23.277190 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 32605
I20260927 07:06:23.277427 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:23.277927 32534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:54907
I20260927 07:06:23.278044 30785 external_mini_cluster.cc:1428] Setting key fb96b3b7d1c5dff5a6a30c100da5faf6
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:23.323841 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33611
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:23.648758 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:23.649086 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:23.649261 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:23.656800 32745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:23.656962 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:23.657120 32745 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:06:23.657210 32745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:23.657361 32745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:23.667485 32745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:23.668896 32745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:23.670459 32745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:23.684087 32751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:23.684108 32750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:23.684744 32745 server_base.cc:1034] running on GCE node
W20260927 07:06:23.684860 32753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:23.685539 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:23.686503 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:23.687757 32745 hybrid_clock.cc:648] HybridClock initialized: now 1790492783687709 us; error 43 us; skew 500 ppm
I20260927 07:06:23.688090 32745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:23 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:23.691377 32745 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:23.692868 32745 webserver.cc:460] Webserver started at http://127.30.16.66:39427/ using document root <none> and password file <none>
I20260927 07:06:23.693564 32745 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:23.693709 32745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:23.694033 32745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:23.696985 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "fd3b5057b1f0416a9febd1c937226e5a"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "abf1d221bd68b96e4b6281677eb4c7f4"
server_key_iv: "d9131d15c858f88daf0c18708bf6ea9e"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.697726 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "fd3b5057b1f0416a9febd1c937226e5a"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "abf1d221bd68b96e4b6281677eb4c7f4"
server_key_iv: "d9131d15c858f88daf0c18708bf6ea9e"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.703047 32745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260927 07:06:23.706663 32760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:23.708052 32745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:23.708233 32745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "fd3b5057b1f0416a9febd1c937226e5a"
format_stamp: "Formatted at 2026-09-27 07:06:23 on dist-test-slave-8vqf"
server_key: "abf1d221bd68b96e4b6281677eb4c7f4"
server_key_iv: "d9131d15c858f88daf0c18708bf6ea9e"
server_key_version: "encryptionkey@0"
I20260927 07:06:23.708441 32745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:23.733618 32745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:23.738214 32745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:23.738525 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:23.741158 32745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:23.741297 32745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:23.741446 32745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:23.741545 32745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:06:23 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492783, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:23.767614 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.742053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:48517 (local address 127.30.16.126:33611)
0927 07:06:23.742226 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:23.742233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:23.743872 (+ 1639us) server_negotiation.cc:407] Connection header received
0927 07:06:23.747498 (+ 3626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:23.747507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:23.747609 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:23.747752 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:23.749581 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.750683 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.753993 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.754632 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.754775 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.760529 (+ 5754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:23.760533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:23.760537 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:23.760572 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:23.763642 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.764877 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.764883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.764886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.765031 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.765706 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.765711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.765716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.766001 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:23.766158 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:23.767029 (+ 871us) server_negotiation.cc:299] Negotiation successful
0927 07:06:23.767248 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:06:23.768815 301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.742745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.66:48517)
0927 07:06:23.743672 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:23.743731 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:23.746614 (+ 2883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:23.748037 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:23.748050 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:23.748731 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:23.749383 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.749408 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.750863 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.750869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.753809 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.753832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.754956 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.754967 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.755380 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.756386 (+ 1006us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:23.756428 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:23.760243 (+ 3815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:23.763815 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.763828 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.763851 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.764708 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.765169 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.765174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.765178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.765545 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.766139 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:23.766147 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:23.766534 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 07:06:23.766942 (+ 408us) client_negotiation.cc:240] Negotiation successful
0927 07:06:23.767516 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":334,"threads_started":1}
I20260927 07:06:23.794281 32745 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:46399
I20260927 07:06:23.794301 411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:46399 every 8 connection(s)
I20260927 07:06:23.795397 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.778092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:59103 (local address 127.30.16.126:33611)
0927 07:06:23.778303 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:23.778311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:23.779505 (+ 1194us) server_negotiation.cc:407] Connection header received
0927 07:06:23.779803 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:23.779812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:23.779920 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:23.780043 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:23.782214 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.783561 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.784985 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.785612 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.785758 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.788138 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:23.788145 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:23.788152 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:23.788218 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:23.792211 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.792907 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.792914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.792919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.793122 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:23.793578 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:23.793586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:23.793592 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:23.794132 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:23.794291 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:23.794817 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 07:06:23.795061 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:06:23.795537 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:23.778190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.66:59103)
0927 07:06:23.779352 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:23.779390 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:23.779570 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:23.780274 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:23.780282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:23.781279 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:23.781972 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.781986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.783747 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.783755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.784807 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:23.784819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:23.785790 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:23.785794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:23.785957 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:23.786726 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:23.786755 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:23.787914 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:23.792353 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.792359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.792363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.792754 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.793259 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:23.793263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:23.793267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:23.793437 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:23.794258 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:23.794263 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:23.794524 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 07:06:23.794725 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 07:06:23.795077 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"thread_start_us":610,"threads_started":1}
I20260927 07:06:23.795851 32745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:23.798014 412 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33611
I20260927 07:06:23.798915 412 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:23.799786 412 heartbeater.cc:511] Master 127.30.16.126:33611 requested a full tablet report, sending...
I20260927 07:06:23.801143 32534 ts_manager.cc:194] Registered new tserver with Master: fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399)
I20260927 07:06:23.801947 32534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:48517
I20260927 07:06:23.802517 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 32745
I20260927 07:06:23.802733 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:23.803172 30785 external_mini_cluster.cc:1428] Setting key 81dbf80b974293446148ab4d549eedde
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:23.846101 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33611
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:40547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:24.173206 416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:24.173498 416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:24.173667 416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:24.181241 416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:24.181365 416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:24.181512 416 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 07:06:24.181591 416 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:06:24.181682 416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:24.191836 416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:24.193202 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:24.194581 416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:24.207930 421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:24.207899 424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:24.207975 422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:24.209134 416 server_base.cc:1034] running on GCE node
I20260927 07:06:24.209753 416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:24.210772 416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:24.211999 416 hybrid_clock.cc:648] HybridClock initialized: now 1790492784211952 us; error 47 us; skew 500 ppm
I20260927 07:06:24.212329 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:24 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:24.215634 416 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:24.217204 416 webserver.cc:460] Webserver started at http://127.30.16.67:41741/ using document root <none> and password file <none>
I20260927 07:06:24.217852 416 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:24.217960 416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:24.218214 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:24.221278 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "3960695285d843fa957588855353b1bc"
format_stamp: "Formatted at 2026-09-27 07:06:24 on dist-test-slave-8vqf"
server_key: "36cfcbd9c2f36162ecbca3bdd09f8951"
server_key_iv: "1d19eac5f66aca21bcc4b99fa92432d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:24.221992 416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "3960695285d843fa957588855353b1bc"
format_stamp: "Formatted at 2026-09-27 07:06:24 on dist-test-slave-8vqf"
server_key: "36cfcbd9c2f36162ecbca3bdd09f8951"
server_key_iv: "1d19eac5f66aca21bcc4b99fa92432d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:24.227255 416 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 07:06:24.230903 431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:24.232405 416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:24.232566 416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "3960695285d843fa957588855353b1bc"
format_stamp: "Formatted at 2026-09-27 07:06:24 on dist-test-slave-8vqf"
server_key: "36cfcbd9c2f36162ecbca3bdd09f8951"
server_key_iv: "1d19eac5f66aca21bcc4b99fa92432d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:24.232749 416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:24.253660 416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:24.258301 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:24.258627 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:24.261574 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:24.261699 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:24.261879 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:24.262014 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:06:24 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:24.281492 32741 heartbeater.cc:503] Master 127.30.16.126:33611 was elected leader, sending a full tablet report...
I20260927 07:06:24.284926 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.262041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:50303 (local address 127.30.16.126:33611)
0927 07:06:24.262322 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.262330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.264109 (+ 1779us) server_negotiation.cc:407] Connection header received
0927 07:06:24.266385 (+ 2276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.266391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.266491 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.266650 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:24.269204 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.270342 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.272533 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.273061 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.273225 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.277307 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:24.277313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:24.277317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:24.277381 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:24.280591 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.281497 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.281505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.281509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.281771 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.282321 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.282328 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.282334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.282805 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:24.282993 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.284061 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.284376 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 07:06:24.285617 440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.262781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.67:50303)
0927 07:06:24.263917 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.263987 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.266049 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.266946 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.266964 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.267880 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:24.268932 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.268963 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.270662 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.270666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.272372 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.272386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.273331 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.273339 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.273586 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.274357 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:24.274394 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:24.277020 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:24.280758 (+ 3738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.280770 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.280789 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.281332 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.281880 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.281887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.281891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.282134 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.283063 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:24.283069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:24.283493 (+ 424us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.283841 (+ 348us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.284338 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":410,"threads_started":1}
I20260927 07:06:24.315078 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.297217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:55947 (local address 127.30.16.126:33611)
0927 07:06:24.297476 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.297486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.298073 (+ 587us) server_negotiation.cc:407] Connection header received
0927 07:06:24.298243 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.298253 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.298364 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.298520 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:24.300068 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.301210 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.303641 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.304249 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.304333 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.307560 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:24.307570 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:24.307578 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:24.307635 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:24.311760 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.312544 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.312552 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.312557 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.312739 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.313295 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.313303 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.313308 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.313728 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:24.313903 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.314364 (+ 461us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.314638 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:06:24.315279 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.297241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.30.16.67:55947)
0927 07:06:24.297966 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.297983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.298114 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.298595 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.298602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.299283 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:24.299852 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.299871 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.301416 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.301425 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.303418 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.303440 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.304755 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.304761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.304893 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.305817 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:24.305839 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:24.307280 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:24.311925 (+ 4645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.311931 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.311936 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.312384 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.312908 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.312917 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.312920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.313084 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.313860 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:24.313865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:24.314079 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.314234 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.314648 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":373,"threads_started":1}
I20260927 07:06:24.329233 416 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:34405
I20260927 07:06:24.329331 550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:34405 every 8 connection(s)
I20260927 07:06:24.330658 416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:24.333019 551 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33611
I20260927 07:06:24.333288 551 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:24.333998 551 heartbeater.cc:511] Master 127.30.16.126:33611 requested a full tablet report, sending...
I20260927 07:06:24.335392 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 416
I20260927 07:06:24.335500 32534 ts_manager.cc:194] Registered new tserver with Master: 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:24.335583 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:24.336129 30785 external_mini_cluster.cc:1428] Setting key 1ce5e1f3e8d94b48c6968997fab5a37b
I20260927 07:06:24.336678 32534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:50303
I20260927 07:06:24.339357 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:06:24 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:06:24 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:24.385613 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.371744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37422 (local address 127.30.16.126:33611)
0927 07:06:24.372022 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.372027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.372261 (+ 234us) server_negotiation.cc:407] Connection header received
0927 07:06:24.372421 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.372425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.372489 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.372583 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:24.373971 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.374834 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.376077 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.376544 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.376674 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.380414 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:24.380419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:24.380423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:24.380458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:24.383403 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.383983 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.383988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.383990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.384135 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.384482 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.384486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.384489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.384773 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:24.384964 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.385182 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.385347 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 07:06:24.390043 32533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:24.397444 32533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:24.420905 564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.409701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46399 (local address 127.0.0.1:47208)
0927 07:06:24.410580 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.410621 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.410810 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.413361 (+ 2551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.413372 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.413457 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:24.413913 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.413925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.415807 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.415815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.418076 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.418097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.419890 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.419897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.420091 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.420147 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.420260 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.420437 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":20,"thread_start_us":386,"threads_started":1}
I20260927 07:06:24.421689 566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.410326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:34405 (local address 127.0.0.1:55366)
0927 07:06:24.411823 (+ 1497us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.411841 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.411952 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.414384 (+ 2432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.414389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.414413 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:24.414814 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.414826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.417176 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.417181 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.419260 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.419275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.420858 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.420864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.421043 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.421078 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.421151 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.421271 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"mutex_wait_us":133,"thread_start_us":1133,"threads_started":1}
I20260927 07:06:24.422143 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.409934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.30.16.66:46399)
0927 07:06:24.412506 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.412518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.412551 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:06:24.412759 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.412771 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.413112 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.413966 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:24.414167 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.415629 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.418244 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.419657 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.421170 (+ 1513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.421378 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.421446 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.421621 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":2218,"threads_started":1}
I20260927 07:06:24.422513 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.411004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.30.16.67:34405)
0927 07:06:24.413948 (+ 2944us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.413954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.413977 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:06:24.414043 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.414047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.414213 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.414356 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:24.415009 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.417037 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.419450 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.420641 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.420789 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.421053 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.421700 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.422084 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2821,"thread_start_us":2650,"threads_started":1}
I20260927 07:06:24.422600 568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.411121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36561 (local address 127.0.0.1:37380)
0927 07:06:24.412809 (+ 1688us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.412825 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.412934 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.415027 (+ 2093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.415031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.415056 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:24.415353 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.415362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.417224 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.417231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.419524 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.419539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.421719 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.421725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.421883 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.421914 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.422006 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.422123 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":71,"thread_start_us":1426,"threads_started":1}
I20260927 07:06:24.422623 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.411828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.30.16.65:36561)
0927 07:06:24.413378 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.413386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.413412 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:24.414677 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.414682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.414842 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.415029 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:24.415523 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.417037 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.419712 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.421210 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.421356 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.421567 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.422071 (+ 504us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.422255 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":1273,"threads_started":1}
I20260927 07:06:24.426036 32675 tablet_service.cc:1467] Processing CreateTablet for tablet 599e65b7284241508ecd6568facbac5d (DEFAULT_TABLE table=test-table [id=27c59d73e9a74a409026dd50594b0cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:24.426172 342 tablet_service.cc:1467] Processing CreateTablet for tablet 599e65b7284241508ecd6568facbac5d (DEFAULT_TABLE table=test-table [id=27c59d73e9a74a409026dd50594b0cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:24.426170 486 tablet_service.cc:1467] Processing CreateTablet for tablet 599e65b7284241508ecd6568facbac5d (DEFAULT_TABLE table=test-table [id=27c59d73e9a74a409026dd50594b0cf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:24.427330 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e65b7284241508ecd6568facbac5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.427493 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e65b7284241508ecd6568facbac5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.427500 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599e65b7284241508ecd6568facbac5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.439316 570 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Bootstrap starting.
I20260927 07:06:24.443099 572 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Bootstrap starting.
I20260927 07:06:24.443171 571 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Bootstrap starting.
I20260927 07:06:24.443454 570 tablet_bootstrap.cc:654] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.444845 570 log.cc:824] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:24.446172 571 tablet_bootstrap.cc:654] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.446568 572 tablet_bootstrap.cc:654] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.447242 571 log.cc:824] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:24.447695 572 log.cc:824] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:24.448078 570 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: No bootstrap required, opened a new log
I20260927 07:06:24.448400 570 ts_tablet_manager.cc:1397] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260927 07:06:24.449683 571 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: No bootstrap required, opened a new log
I20260927 07:06:24.449970 571 ts_tablet_manager.cc:1397] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 07:06:24.457141 572 tablet_bootstrap.cc:492] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: No bootstrap required, opened a new log
I20260927 07:06:24.457417 572 ts_tablet_manager.cc:1397] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260927 07:06:24.457670 571 raft_consensus.cc:348] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.458032 571 raft_consensus.cc:374] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.458153 571 raft_consensus.cc:729] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.458775 571 consensus_queue.cc:260] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.459805 412 heartbeater.cc:503] Master 127.30.16.126:33611 was elected leader, sending a full tablet report...
I20260927 07:06:24.460484 571 ts_tablet_manager.cc:1428] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 07:06:24.460233 570 raft_consensus.cc:348] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.460624 570 raft_consensus.cc:374] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.460709 570 raft_consensus.cc:729] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb54f3198c084ad29daaa89edbfcb954, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.461392 570 consensus_queue.cc:260] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.465620 570 ts_tablet_manager.cc:1428] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20260927 07:06:24.467299 572 raft_consensus.cc:348] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.467815 572 raft_consensus.cc:374] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.467937 572 raft_consensus.cc:729] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960695285d843fa957588855353b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.468551 572 consensus_queue.cc:260] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.469794 551 heartbeater.cc:503] Master 127.30.16.126:33611 was elected leader, sending a full tablet report...
I20260927 07:06:24.470321 572 ts_tablet_manager.cc:1428] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260927 07:06:24.523218 32742 tablet.cc:2369] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:24.548003 413 tablet.cc:2369] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:24.582299 552 tablet.cc:2369] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:06:24.586349 576 raft_consensus.cc:482] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:24.586696 576 raft_consensus.cc:504] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.588470 576 leader_election.cc:288] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:24.595366 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.588881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52117 (local address 127.30.16.65:36561)
0927 07:06:24.589119 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.589129 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.590071 (+ 942us) server_negotiation.cc:407] Connection header received
0927 07:06:24.590238 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.590243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.590322 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.590415 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:24.591114 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.591795 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.593639 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.594410 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.594618 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.594696 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.594995 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.595124 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:06:24.595391 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.589085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:38271 (local address 127.30.16.67:34405)
0927 07:06:24.589324 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.589333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.589982 (+ 649us) server_negotiation.cc:407] Connection header received
0927 07:06:24.590191 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.590196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.590265 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.590361 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:24.591054 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.591766 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.593639 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.594384 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.594587 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.594667 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.594999 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.595138 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 07:06:24.595597 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.588892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36561 (local address 127.30.16.66:52117)
0927 07:06:24.589969 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.589988 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.590094 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.590444 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.590449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.590479 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:24.590942 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.590952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.591929 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.591936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.593494 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.593505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.594659 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.594665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.594820 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.594858 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.594968 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.595111 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"mutex_wait_us":52,"spinlock_wait_cycles":85888,"thread_start_us":670,"threads_started":1}
I20260927 07:06:24.595640 579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.588820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:34405 (local address 127.30.16.66:38271)
0927 07:06:24.589874 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.589890 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.590064 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.590383 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.590388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.590410 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:24.590903 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.590915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.591902 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.591910 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.593484 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.593495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.594633 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.594644 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.594822 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.594871 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.594974 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.595126 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":82,"spinlock_wait_cycles":23808,"thread_start_us":492,"threads_started":1}
I20260927 07:06:24.596632 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e65b7284241508ecd6568facbac5d" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954" is_pre_election: true
I20260927 07:06:24.596629 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e65b7284241508ecd6568facbac5d" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc" is_pre_election: true
I20260927 07:06:24.597105 32696 raft_consensus.cc:2454] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:24.597110 506 raft_consensus.cc:2454] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:24.597813 32764 leader_election.cc:302] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, bb54f3198c084ad29daaa89edbfcb954, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:24.598272 576 raft_consensus.cc:2781] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:24.598412 576 raft_consensus.cc:482] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:24.598590 576 raft_consensus.cc:3037] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.600713 576 raft_consensus.cc:504] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.601361 576 leader_election.cc:288] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Requested vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:24.601897 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e65b7284241508ecd6568facbac5d" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954"
I20260927 07:06:24.601935 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599e65b7284241508ecd6568facbac5d" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc"
I20260927 07:06:24.602103 32696 raft_consensus.cc:3037] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.602136 506 raft_consensus.cc:3037] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.604544 506 raft_consensus.cc:2454] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:24.604596 32696 raft_consensus.cc:2454] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:24.604949 32764 leader_election.cc:302] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, bb54f3198c084ad29daaa89edbfcb954, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:24.605260 576 raft_consensus.cc:2781] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:24.605643 576 raft_consensus.cc:686] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Becoming Leader. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Running, Role: LEADER
I20260927 07:06:24.606197 576 consensus_queue.cc:237] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.610610 32534 catalog_manager.cc:5495] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66). New cstate: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } health_report { overall_health: UNKNOWN } } }
Sep 27 07:06:24 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492782, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:24.694708 32629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.681245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37434 (local address 127.30.16.126:33611)
0927 07:06:24.681453 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.681460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.682169 (+ 709us) server_negotiation.cc:407] Connection header received
0927 07:06:24.682347 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.682351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.682427 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.682524 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:24.683998 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.684953 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.686178 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.686737 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.686838 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.690105 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:24.690110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:24.690113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:24.690140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:24.692396 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.693025 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.693029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.693031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.693141 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:24.693464 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:24.693467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:24.693470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:24.693720 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:24.693821 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.694203 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.694422 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":75}
I20260927 07:06:24.694727 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.681374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33611 (local address 127.0.0.1:37434)
0927 07:06:24.682027 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.682043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.682227 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.682626 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.682629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.683427 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:24.683820 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.683855 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.685063 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.685066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.686028 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.686038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.686899 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.686904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.687025 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.687717 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:24.687734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:24.689935 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:24.692514 (+ 2579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.692524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.692530 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.692928 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.693241 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:24.693244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:24.693246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:24.693379 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:24.693820 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:24.693826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:24.693969 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.694099 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.694291 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":352,"threads_started":1}
I20260927 07:06:24.699558 32534 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37434:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:06:24.700512 32534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:24.707192 32675 tablet_service.cc:1467] Processing CreateTablet for tablet 67e3c1514e144272a4947d5da3db73c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29263d192232421092cb268e5fa75288]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:24.707435 486 tablet_service.cc:1467] Processing CreateTablet for tablet 67e3c1514e144272a4947d5da3db73c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29263d192232421092cb268e5fa75288]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:24.707391 342 tablet_service.cc:1467] Processing CreateTablet for tablet 67e3c1514e144272a4947d5da3db73c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=29263d192232421092cb268e5fa75288]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:06:24.708244 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e3c1514e144272a4947d5da3db73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.708245 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e3c1514e144272a4947d5da3db73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.708287 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e3c1514e144272a4947d5da3db73c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.714789 571 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: Bootstrap starting.
I20260927 07:06:24.715389 572 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: Bootstrap starting.
I20260927 07:06:24.715466 570 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: Bootstrap starting.
I20260927 07:06:24.716763 571 tablet_bootstrap.cc:654] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.717044 572 tablet_bootstrap.cc:654] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.717288 570 tablet_bootstrap.cc:654] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.719310 571 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: No bootstrap required, opened a new log
I20260927 07:06:24.719419 572 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: No bootstrap required, opened a new log
I20260927 07:06:24.719493 571 ts_tablet_manager.cc:1397] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 07:06:24.719578 572 ts_tablet_manager.cc:1397] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260927 07:06:24.720027 570 tablet_bootstrap.cc:492] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: No bootstrap required, opened a new log
I20260927 07:06:24.720182 570 ts_tablet_manager.cc:1397] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260927 07:06:24.720220 571 raft_consensus.cc:348] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.720476 571 raft_consensus.cc:374] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.720575 571 raft_consensus.cc:729] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.720432 572 raft_consensus.cc:348] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.720685 572 raft_consensus.cc:374] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.720798 572 raft_consensus.cc:729] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960695285d843fa957588855353b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.720810 571 consensus_queue.cc:260] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.721016 570 raft_consensus.cc:348] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.721089 572 consensus_queue.cc:260] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.721276 570 raft_consensus.cc:374] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:24.721374 570 raft_consensus.cc:729] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb54f3198c084ad29daaa89edbfcb954, State: Initialized, Role: FOLLOWER
I20260927 07:06:24.721387 571 ts_tablet_manager.cc:1428] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:24.721815 572 ts_tablet_manager.cc:1428] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 07:06:24.721690 570 consensus_queue.cc:260] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.721604 578 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.721994 578 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.722237 577 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.722692 577 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.723234 570 ts_tablet_manager.cc:1428] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:24.723121 576 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.723405 576 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.786195 576 raft_consensus.cc:482] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:24.786460 576 raft_consensus.cc:504] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.787341 576 leader_election.cc:288] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:24.787796 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e3c1514e144272a4947d5da3db73c0" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954" is_pre_election: true
I20260927 07:06:24.787923 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e3c1514e144272a4947d5da3db73c0" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc" is_pre_election: true
I20260927 07:06:24.788051 32696 raft_consensus.cc:2454] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:24.788237 506 raft_consensus.cc:2454] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:24.788502 32763 leader_election.cc:302] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb54f3198c084ad29daaa89edbfcb954, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:24.788791 576 raft_consensus.cc:2781] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:24.788899 576 raft_consensus.cc:482] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:24.789009 576 raft_consensus.cc:3037] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.791075 576 raft_consensus.cc:504] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.791644 576 leader_election.cc:288] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Requested vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:24.792073 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e3c1514e144272a4947d5da3db73c0" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954"
I20260927 07:06:24.792161 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e3c1514e144272a4947d5da3db73c0" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc"
I20260927 07:06:24.792300 32696 raft_consensus.cc:3037] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.792387 506 raft_consensus.cc:3037] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:24.794432 506 raft_consensus.cc:2454] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:24.794818 32696 raft_consensus.cc:2454] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:24.794930 32764 leader_election.cc:302] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:24.795238 576 raft_consensus.cc:2781] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:24.795658 576 raft_consensus.cc:686] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Becoming Leader. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Running, Role: LEADER
I20260927 07:06:24.795936 576 consensus_queue.cc:237] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:24.797797 581 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: TxnStatusTablet state changed. Reason: New leader fd3b5057b1f0416a9febd1c937226e5a. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.798198 581 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:24.799144 593 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:06:24.799433 593 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:06:24.799453 32533 catalog_manager.cc:5495] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66). New cstate: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:24.822198 588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.815970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46399 (local address 127.0.0.1:47222)
0927 07:06:24.816213 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.816233 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.816373 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.816835 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.816838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.816847 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:24.816999 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.817005 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.818395 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.818399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.819453 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.819464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.820451 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.820457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.820661 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.820733 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:24.821759 (+ 1026us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:24.821784 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.821853 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.821972 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 07:06:24.822264 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.815966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47222 (local address 127.30.16.66:46399)
0927 07:06:24.816179 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.816186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.816325 (+ 139us) server_negotiation.cc:407] Connection header received
0927 07:06:24.816554 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.816560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.816665 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.816819 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:24.817123 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.818270 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.819589 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.820244 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.820375 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.820934 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:24.821642 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:24.821791 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.821897 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.821984 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:06:24.830201 506 raft_consensus.cc:1261] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:24.830219 32696 raft_consensus.cc:1261] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:24.831269 581 consensus_queue.cc:1035] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:24.831615 576 consensus_queue.cc:1035] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:24.848433 578 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: TxnStatusTablet state changed. Reason: New leader fd3b5057b1f0416a9febd1c937226e5a. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.848716 578 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.849442 577 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: TxnStatusTablet state changed. Reason: New leader fd3b5057b1f0416a9febd1c937226e5a. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.849720 577 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.856366 578 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.856603 578 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P 3960695285d843fa957588855353b1bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.858780 581 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.859059 581 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:24.859853 577 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.860088 577 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P bb54f3198c084ad29daaa89edbfcb954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:06:24.860225 581 tablet_replica.cc:439] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } } }
I20260927 07:06:24.860440 581 tablet_replica.cc:442] T 67e3c1514e144272a4947d5da3db73c0 P fd3b5057b1f0416a9febd1c937226e5a: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:06:24.872655 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.866804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47232 (local address 127.30.16.66:46399)
0927 07:06:24.866976 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.866980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.867224 (+ 244us) server_negotiation.cc:407] Connection header received
0927 07:06:24.867366 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.867371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.867454 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.867550 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:24.867895 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.869087 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.870488 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.871269 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.871405 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.871866 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:24.872023 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:24.872126 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.872231 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.872314 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 07:06:24.891932 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.885251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43737 (local address 127.30.16.66:46399)
0927 07:06:24.885456 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:24.885464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:24.886090 (+ 626us) server_negotiation.cc:407] Connection header received
0927 07:06:24.886249 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:24.886255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:24.886337 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:24.886439 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:24.887050 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.888167 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.889790 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.890606 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.890737 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.891170 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:24.891294 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:24.891384 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:24.891507 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 07:06:24.891584 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":57}
I20260927 07:06:24.891950 607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:24.885242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46399 (local address 127.30.16.66:43737)
0927 07:06:24.885988 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:24.886005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:24.886129 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:24.886456 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:24.886459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:24.886468 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:24.886899 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.886908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.888281 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.888285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.889658 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:24.889669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:24.890786 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:24.890791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:24.890902 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:24.890955 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:24.891393 (+ 438us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:24.891400 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:06:24.891460 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:06:24.891568 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":70,"thread_start_us":404,"threads_started":1}
I20260927 07:06:24.902276 32696 raft_consensus.cc:1261] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:24.902611 506 raft_consensus.cc:1261] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:24.902966 608 consensus_queue.cc:1035] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:24.903452 608 consensus_queue.cc:1035] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:24.912510 596 mvcc.cc:205] Tried to move back new op lower bound from 7333858446952288256 to 7333858445750976512. Current Snapshot: MvccSnapshot[applied={T|T < 7333858446952288256}]
I20260927 07:06:24.969328 32533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:24.969602 32533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:24.973417 32533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Sending DeleteTablet for 3 replicas of tablet 599e65b7284241508ecd6568facbac5d
I20260927 07:06:24.974519 342 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e65b7284241508ecd6568facbac5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:24 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47208
I20260927 07:06:24.974676 486 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e65b7284241508ecd6568facbac5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:24 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55366
I20260927 07:06:24.974670 32675 tablet_service.cc:1514] Processing DeleteTablet for tablet 599e65b7284241508ecd6568facbac5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:24 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37380
I20260927 07:06:24.978049 32533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:24.978912 32533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:24.985733 616 tablet_replica.cc:330] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: stopping tablet replica
I20260927 07:06:24.985981 614 tablet_replica.cc:330] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: stopping tablet replica
I20260927 07:06:24.986624 614 raft_consensus.cc:2229] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:24.987191 616 raft_consensus.cc:2229] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:24.987416 614 raft_consensus.cc:2258] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:24.988164 616 raft_consensus.cc:2258] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:24.988853 32670 tablet_service.cc:1467] Processing CreateTablet for tablet fe160407b9ff411090132db70af0b193 (DEFAULT_TABLE table=test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:24.989130 341 tablet_service.cc:1467] Processing CreateTablet for tablet fe160407b9ff411090132db70af0b193 (DEFAULT_TABLE table=test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:24.989380 32670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe160407b9ff411090132db70af0b193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.989601 341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe160407b9ff411090132db70af0b193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:24.990266 614 ts_tablet_manager.cc:1905] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:24.995751 571 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Bootstrap starting.
I20260927 07:06:24.996580 570 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Bootstrap starting.
I20260927 07:06:24.997637 571 tablet_bootstrap.cc:654] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.998369 614 ts_tablet_manager.cc:1918] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:24.998586 614 log.cc:1196] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/599e65b7284241508ecd6568facbac5d
I20260927 07:06:24.998834 570 tablet_bootstrap.cc:654] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:24.999156 614 ts_tablet_manager.cc:1939] T 599e65b7284241508ecd6568facbac5d P fd3b5057b1f0416a9febd1c937226e5a: Deleting consensus metadata
I20260927 07:06:25.001129 32520 catalog_manager.cc:4841] TS fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399): tablet 599e65b7284241508ecd6568facbac5d (table test-table [id=27c59d73e9a74a409026dd50594b0cf4]) successfully deleted
I20260927 07:06:24.988164 486 tablet_service.cc:1467] Processing CreateTablet for tablet fe160407b9ff411090132db70af0b193 (DEFAULT_TABLE table=test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:25.002524 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe160407b9ff411090132db70af0b193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:25.003006 570 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: No bootstrap required, opened a new log
I20260927 07:06:25.003221 570 ts_tablet_manager.cc:1397] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 07:06:25.003997 570 raft_consensus.cc:348] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.004349 570 raft_consensus.cc:374] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:25.004472 570 raft_consensus.cc:729] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb54f3198c084ad29daaa89edbfcb954, State: Initialized, Role: FOLLOWER
I20260927 07:06:25.004791 570 consensus_queue.cc:260] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.008025 570 ts_tablet_manager.cc:1428] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 07:06:25.009110 572 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Bootstrap starting.
I20260927 07:06:25.010929 572 tablet_bootstrap.cc:654] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:25.012212 616 ts_tablet_manager.cc:1905] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:25.014024 572 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: No bootstrap required, opened a new log
I20260927 07:06:25.014194 572 ts_tablet_manager.cc:1397] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 07:06:25.015074 572 raft_consensus.cc:348] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.015327 572 raft_consensus.cc:374] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:25.015409 572 raft_consensus.cc:729] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960695285d843fa957588855353b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:06:25.015501 615 tablet_replica.cc:330] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: stopping tablet replica
I20260927 07:06:25.015703 572 consensus_queue.cc:260] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.016007 615 raft_consensus.cc:2229] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:25.016373 615 raft_consensus.cc:2258] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:25.018468 615 ts_tablet_manager.cc:1905] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:25.019814 572 ts_tablet_manager.cc:1428] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260927 07:06:25.021515 616 ts_tablet_manager.cc:1918] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:25.021726 616 log.cc:1196] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/599e65b7284241508ecd6568facbac5d
I20260927 07:06:25.022300 616 ts_tablet_manager.cc:1939] T 599e65b7284241508ecd6568facbac5d P 3960695285d843fa957588855353b1bc: Deleting consensus metadata
I20260927 07:06:25.023335 571 tablet_bootstrap.cc:492] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: No bootstrap required, opened a new log
I20260927 07:06:25.023523 571 ts_tablet_manager.cc:1397] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20260927 07:06:25.024328 32521 catalog_manager.cc:4841] TS 3960695285d843fa957588855353b1bc (127.30.16.67:34405): tablet 599e65b7284241508ecd6568facbac5d (table test-table [id=27c59d73e9a74a409026dd50594b0cf4]) successfully deleted
I20260927 07:06:25.024328 571 raft_consensus.cc:348] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.024576 571 raft_consensus.cc:374] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:25.024662 571 raft_consensus.cc:729] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Initialized, Role: FOLLOWER
I20260927 07:06:25.024945 571 consensus_queue.cc:260] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.025712 615 ts_tablet_manager.cc:1918] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:25.025882 615 log.cc:1196] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/599e65b7284241508ecd6568facbac5d
I20260927 07:06:25.026389 615 ts_tablet_manager.cc:1939] T 599e65b7284241508ecd6568facbac5d P bb54f3198c084ad29daaa89edbfcb954: Deleting consensus metadata
I20260927 07:06:25.026504 571 ts_tablet_manager.cc:1428] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 07:06:25.027886 32520 catalog_manager.cc:4841] TS bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561): tablet 599e65b7284241508ecd6568facbac5d (table test-table [id=27c59d73e9a74a409026dd50594b0cf4]) successfully deleted
I20260927 07:06:25.067541 578 raft_consensus.cc:482] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:25.067833 578 raft_consensus.cc:504] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.069545 578 leader_election.cc:288] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399)
I20260927 07:06:25.079200 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.070842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:37405 (local address 127.30.16.66:46399)
0927 07:06:25.071017 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:25.071022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:25.072795 (+ 1773us) server_negotiation.cc:407] Connection header received
0927 07:06:25.073029 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:25.073033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:25.073094 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:25.073195 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:25.073771 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.074845 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.077241 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.078150 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.078328 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.078403 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:25.078786 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 07:06:25.078924 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:06:25.079278 621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.070631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46399 (local address 127.30.16.67:37405)
0927 07:06:25.072678 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:25.072693 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:25.072841 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:25.073208 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:25.073213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:25.073235 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:25.073576 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.073586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.075026 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.075031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.077097 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.077108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.078339 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.078345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.078540 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.078615 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 07:06:25.078718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:06:25.078882 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"thread_start_us":1782,"threads_started":1}
I20260927 07:06:25.079871 620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.069765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36561 (local address 127.30.16.67:48309)
0927 07:06:25.070453 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:25.070474 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:25.070731 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:25.071538 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:25.071546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:25.071584 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:25.072048 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.072064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.074638 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.074643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.076939 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.076954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.078619 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.078625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.078850 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.078879 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 07:06:25.078962 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 07:06:25.079099 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":405,"threads_started":1}
I20260927 07:06:25.080207 363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe160407b9ff411090132db70af0b193" candidate_uuid: "3960695285d843fa957588855353b1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b5057b1f0416a9febd1c937226e5a" is_pre_election: true
I20260927 07:06:25.080502 569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.069874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:48309 (local address 127.30.16.65:36561)
0927 07:06:25.070062 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:25.070067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:25.070590 (+ 523us) server_negotiation.cc:407] Connection header received
0927 07:06:25.070946 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:25.070954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:25.071081 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:25.071158 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:25.072269 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.074118 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.077119 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.078349 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.078510 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.078599 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:25.080083 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0927 07:06:25.080230 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:06:25.080695 363 raft_consensus.cc:2454] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960695285d843fa957588855353b1bc in term 0.
I20260927 07:06:25.081389 434 leader_election.cc:302] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:25.081431 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe160407b9ff411090132db70af0b193" candidate_uuid: "3960695285d843fa957588855353b1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954" is_pre_election: true
I20260927 07:06:25.081670 32696 raft_consensus.cc:2454] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960695285d843fa957588855353b1bc in term 0.
I20260927 07:06:25.081790 578 raft_consensus.cc:2781] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:25.081915 578 raft_consensus.cc:482] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:25.082041 578 raft_consensus.cc:3037] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:25.084302 578 raft_consensus.cc:504] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.084841 578 leader_election.cc:288] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [CANDIDATE]: Term 1 election: Requested vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399)
I20260927 07:06:25.085489 363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe160407b9ff411090132db70af0b193" candidate_uuid: "3960695285d843fa957588855353b1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd3b5057b1f0416a9febd1c937226e5a"
I20260927 07:06:25.085693 363 raft_consensus.cc:3037] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:25.085793 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe160407b9ff411090132db70af0b193" candidate_uuid: "3960695285d843fa957588855353b1bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954"
I20260927 07:06:25.086011 32696 raft_consensus.cc:3037] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:25.088171 32696 raft_consensus.cc:2454] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960695285d843fa957588855353b1bc in term 1.
I20260927 07:06:25.088642 434 leader_election.cc:302] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, bb54f3198c084ad29daaa89edbfcb954; no voters:
I20260927 07:06:25.088994 578 raft_consensus.cc:2781] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:25.089030 363 raft_consensus.cc:2454] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960695285d843fa957588855353b1bc in term 1.
I20260927 07:06:25.089556 578 raft_consensus.cc:686] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 1 LEADER]: Becoming Leader. State: Replica: 3960695285d843fa957588855353b1bc, State: Running, Role: LEADER
I20260927 07:06:25.089943 578 consensus_queue.cc:237] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:25.093490 32533 catalog_manager.cc:5495] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3960695285d843fa957588855353b1bc (127.30.16.67). New cstate: current_term: 1 leader_uuid: "3960695285d843fa957588855353b1bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } health_report { overall_health: HEALTHY } } }
I20260927 07:06:25.139369 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.132779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55380 (local address 127.30.16.67:34405)
0927 07:06:25.133264 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:25.133268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:25.133286 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:06:25.133471 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:25.133475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:25.133572 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:25.133712 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:25.134123 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.135035 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.136881 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.137370 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.137469 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.138062 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:25.138678 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:25.138787 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:25.138943 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 07:06:25.139010 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":283,"threads_started":1}
I20260927 07:06:25.158308 627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.150360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46399 (local address 127.30.16.67:58733)
0927 07:06:25.152581 (+ 2221us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:25.152611 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:25.152730 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:25.153087 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:25.153090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:25.153099 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:25.153382 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.153389 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.154843 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.154847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.156109 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.156120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.157195 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.157200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.157316 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.157371 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:25.157828 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:25.157835 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:06:25.157915 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 07:06:25.158024 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1909,"threads_started":1}
I20260927 07:06:25.158383 565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.150385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:58733 (local address 127.30.16.66:46399)
0927 07:06:25.150619 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:25.150626 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:25.152708 (+ 2082us) server_negotiation.cc:407] Connection header received
0927 07:06:25.152849 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:25.152855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:25.152958 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:25.153063 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:25.153532 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.154706 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.156281 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.157011 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.157164 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.157564 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:25.157719 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:25.157838 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:25.157962 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 07:06:25.158042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:06:25.167049 32696 raft_consensus.cc:1261] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Refusing update from remote peer 3960695285d843fa957588855353b1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:25.167253 363 raft_consensus.cc:1261] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Refusing update from remote peer 3960695285d843fa957588855353b1bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:25.167812 622 consensus_queue.cc:1035] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:25.168303 578 consensus_queue.cc:1035] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:25.173377 611 mvcc.cc:205] Tried to move back new op lower bound from 7333858448037531648 to 7333858447732350976. Current Snapshot: MvccSnapshot[applied={T|T < 7333858448037531648}]
I20260927 07:06:25.213901 607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.207250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:34405 (local address 127.30.16.66:44591)
0927 07:06:25.207479 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:25.207500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:25.207618 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:25.208158 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:25.208164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:25.208177 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:06:25.208389 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.208401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.209835 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.209843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.211741 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:25.211811 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.212816 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.212820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:25.212942 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.212986 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:06:25.213435 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:06:25.213470 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:06:25.213538 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 07:06:25.213660 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 07:06:25.214001 624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:25.207172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:44591 (local address 127.30.16.67:34405)
0927 07:06:25.207484 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:25.207497 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:25.207733 (+ 236us) server_negotiation.cc:407] Connection header received
0927 07:06:25.207796 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:25.207801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:25.207896 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:25.207985 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:25.208579 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.209673 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.211993 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:25.212660 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:25.212815 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:25.213151 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:25.213295 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:25.213397 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:25.213591 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 07:06:25.213679 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
W20260927 07:06:25.215195 611 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 07:06:26.841245 32532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:26.841491 32532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:26.845636 32532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Sending DeleteTablet for 3 replicas of tablet fe160407b9ff411090132db70af0b193
I20260927 07:06:26.846501 32675 tablet_service.cc:1514] Processing DeleteTablet for tablet fe160407b9ff411090132db70af0b193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:26 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37380
I20260927 07:06:26.846588 486 tablet_service.cc:1514] Processing DeleteTablet for tablet fe160407b9ff411090132db70af0b193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:26 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55366
I20260927 07:06:26.846694 342 tablet_service.cc:1514] Processing DeleteTablet for tablet fe160407b9ff411090132db70af0b193 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:26 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47208
I20260927 07:06:26.857928 660 tablet_replica.cc:330] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: stopping tablet replica
I20260927 07:06:26.858145 659 tablet_replica.cc:330] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: stopping tablet replica
I20260927 07:06:26.858242 660 raft_consensus.cc:2229] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:26.858397 661 tablet_replica.cc:330] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: stopping tablet replica
I20260927 07:06:26.858408 659 raft_consensus.cc:2229] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:26.858740 659 raft_consensus.cc:2258] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:26.858753 661 raft_consensus.cc:2229] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:26.858963 660 raft_consensus.cc:2258] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:26.859411 661 raft_consensus.cc:2258] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:26.859838 659 ts_tablet_manager.cc:1905] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:26.860289 660 ts_tablet_manager.cc:1905] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:26.860461 661 ts_tablet_manager.cc:1905] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:26.866772 659 ts_tablet_manager.cc:1918] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:26.866772 660 ts_tablet_manager.cc:1918] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:26.866910 660 log.cc:1196] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/fe160407b9ff411090132db70af0b193
I20260927 07:06:26.866993 659 log.cc:1196] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/fe160407b9ff411090132db70af0b193
I20260927 07:06:26.867477 660 ts_tablet_manager.cc:1939] T fe160407b9ff411090132db70af0b193 P 3960695285d843fa957588855353b1bc: Deleting consensus metadata
I20260927 07:06:26.867506 659 ts_tablet_manager.cc:1939] T fe160407b9ff411090132db70af0b193 P bb54f3198c084ad29daaa89edbfcb954: Deleting consensus metadata
I20260927 07:06:26.867976 661 ts_tablet_manager.cc:1918] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:26.868155 661 log.cc:1196] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/fe160407b9ff411090132db70af0b193
I20260927 07:06:26.868703 661 ts_tablet_manager.cc:1939] T fe160407b9ff411090132db70af0b193 P fd3b5057b1f0416a9febd1c937226e5a: Deleting consensus metadata
I20260927 07:06:26.868656 32521 catalog_manager.cc:4841] TS 3960695285d843fa957588855353b1bc (127.30.16.67:34405): tablet fe160407b9ff411090132db70af0b193 (table test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]) successfully deleted
I20260927 07:06:26.868656 32520 catalog_manager.cc:4841] TS bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561): tablet fe160407b9ff411090132db70af0b193 (table test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]) successfully deleted
I20260927 07:06:26.869707 32520 catalog_manager.cc:4841] TS fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399): tablet fe160407b9ff411090132db70af0b193 (table test-table [id=2c3ee7c3ad114a07bd3fff001d560b5c]) successfully deleted
Sep 27 07:06:26 dist-test-slave-8vqf krb5kdc[32489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:06:26 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:26.909703 669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:26.890076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.30.16.126:33611)
0927 07:06:26.890733 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:26.890738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:26.890838 (+ 100us) server_negotiation.cc:407] Connection header received
0927 07:06:26.891121 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:26.891126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:26.891189 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:26.891306 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:26.892685 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:26.893458 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:26.894855 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:26.895257 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:26.895435 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:26.904228 (+ 8793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:26.904234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:26.904238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:26.904282 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:26.906917 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:26.907657 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:26.907662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:26.907665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:26.907819 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:26.908241 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:26.908245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:26.908247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:26.908540 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:26.908782 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:26.909094 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 07:06:26.909329 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":345,"threads_started":1}
I20260927 07:06:26.913920 32532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:26.914608 32532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:26.921470 32675 tablet_service.cc:1467] Processing CreateTablet for tablet 596932cfa4ac4a62afcac553094fe057 (DEFAULT_TABLE table=test-table [id=24f45903c903455cb355d3e1bae25e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:26.921613 342 tablet_service.cc:1467] Processing CreateTablet for tablet 596932cfa4ac4a62afcac553094fe057 (DEFAULT_TABLE table=test-table [id=24f45903c903455cb355d3e1bae25e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:26.921767 486 tablet_service.cc:1467] Processing CreateTablet for tablet 596932cfa4ac4a62afcac553094fe057 (DEFAULT_TABLE table=test-table [id=24f45903c903455cb355d3e1bae25e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:26.921957 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596932cfa4ac4a62afcac553094fe057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:26.922029 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596932cfa4ac4a62afcac553094fe057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:26.922163 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 596932cfa4ac4a62afcac553094fe057. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:26.928182 670 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Bootstrap starting.
I20260927 07:06:26.928871 671 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Bootstrap starting.
I20260927 07:06:26.929709 672 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Bootstrap starting.
I20260927 07:06:26.930217 670 tablet_bootstrap.cc:654] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:26.930991 671 tablet_bootstrap.cc:654] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:26.931504 672 tablet_bootstrap.cc:654] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:26.933112 670 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: No bootstrap required, opened a new log
I20260927 07:06:26.933301 670 ts_tablet_manager.cc:1397] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 07:06:26.933612 672 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: No bootstrap required, opened a new log
I20260927 07:06:26.933775 672 ts_tablet_manager.cc:1397] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 07:06:26.934355 672 raft_consensus.cc:348] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.934362 670 raft_consensus.cc:348] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.934723 670 raft_consensus.cc:374] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:26.934746 672 raft_consensus.cc:374] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:26.934844 670 raft_consensus.cc:729] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Initialized, Role: FOLLOWER
I20260927 07:06:26.935037 672 raft_consensus.cc:729] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960695285d843fa957588855353b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:06:26.935129 670 consensus_queue.cc:260] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.935376 672 consensus_queue.cc:260] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.936115 672 ts_tablet_manager.cc:1428] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:26.936672 670 ts_tablet_manager.cc:1428] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 07:06:26.938138 671 tablet_bootstrap.cc:492] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: No bootstrap required, opened a new log
I20260927 07:06:26.938309 671 ts_tablet_manager.cc:1397] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 07:06:26.939616 671 raft_consensus.cc:348] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.939858 671 raft_consensus.cc:374] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:26.939966 671 raft_consensus.cc:729] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb54f3198c084ad29daaa89edbfcb954, State: Initialized, Role: FOLLOWER
I20260927 07:06:26.940236 671 consensus_queue.cc:260] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.941094 671 ts_tablet_manager.cc:1428] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 07:06:26.979044 643 raft_consensus.cc:482] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:26.979327 643 raft_consensus.cc:504] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.980141 643 leader_election.cc:288] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:26.980722 32695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596932cfa4ac4a62afcac553094fe057" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954" is_pre_election: true
I20260927 07:06:26.980743 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596932cfa4ac4a62afcac553094fe057" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc" is_pre_election: true
I20260927 07:06:26.980973 32695 raft_consensus.cc:2454] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:26.980980 506 raft_consensus.cc:2454] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:26.981459 32763 leader_election.cc:302] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bb54f3198c084ad29daaa89edbfcb954, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:26.981801 643 raft_consensus.cc:2781] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:26.981958 643 raft_consensus.cc:482] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:26.982080 643 raft_consensus.cc:3037] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:26.984261 643 raft_consensus.cc:504] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.984763 643 leader_election.cc:288] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Requested vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:26.985256 32695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596932cfa4ac4a62afcac553094fe057" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954"
I20260927 07:06:26.985301 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "596932cfa4ac4a62afcac553094fe057" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc"
I20260927 07:06:26.985461 32695 raft_consensus.cc:3037] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:26.985500 506 raft_consensus.cc:3037] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:26.988019 32695 raft_consensus.cc:2454] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:26.988019 506 raft_consensus.cc:2454] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:26.988454 32763 leader_election.cc:302] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, bb54f3198c084ad29daaa89edbfcb954, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:26.988776 643 raft_consensus.cc:2781] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:26.988945 643 raft_consensus.cc:686] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Becoming Leader. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Running, Role: LEADER
I20260927 07:06:26.989226 643 consensus_queue.cc:237] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:26.991828 32533 catalog_manager.cc:5495] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66). New cstate: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:27.021238 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:27.016120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.30.16.66:46399)
0927 07:06:27.016680 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:27.016684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:27.016703 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:06:27.016756 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:27.016761 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:27.016823 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:27.016909 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:27.017252 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:27.018008 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:27.019343 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:27.019762 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:27.019853 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:27.020232 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:27.020350 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:27.020436 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:27.020680 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 07:06:27.020754 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":340,"threads_started":1}
I20260927 07:06:27.030375 32695 raft_consensus.cc:1261] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:27.030391 506 raft_consensus.cc:1261] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:27.030964 634 consensus_queue.cc:1035] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:27.031280 643 consensus_queue.cc:1035] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:27.074879 32532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:27.075134 32532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:27.077967 32532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Sending DeleteTablet for 3 replicas of tablet 596932cfa4ac4a62afcac553094fe057
I20260927 07:06:27.078855 32675 tablet_service.cc:1514] Processing DeleteTablet for tablet 596932cfa4ac4a62afcac553094fe057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37380
I20260927 07:06:27.078869 486 tablet_service.cc:1514] Processing DeleteTablet for tablet 596932cfa4ac4a62afcac553094fe057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55366
I20260927 07:06:27.079138 659 tablet_replica.cc:330] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: stopping tablet replica
I20260927 07:06:27.079140 660 tablet_replica.cc:330] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: stopping tablet replica
I20260927 07:06:27.079355 660 raft_consensus.cc:2229] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:27.079365 659 raft_consensus.cc:2229] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:27.079562 659 raft_consensus.cc:2258] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:27.079602 660 raft_consensus.cc:2258] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:27.079952 32532 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 07:06:27.080631 659 ts_tablet_manager.cc:1905] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 07:06:27.080688 32532 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:27.080891 660 ts_tablet_manager.cc:1905] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:27.086772 342 tablet_service.cc:1514] Processing DeleteTablet for tablet 596932cfa4ac4a62afcac553094fe057 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47208
I20260927 07:06:27.087016 659 ts_tablet_manager.cc:1918] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:27.087203 659 log.cc:1196] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/596932cfa4ac4a62afcac553094fe057
I20260927 07:06:27.087281 660 ts_tablet_manager.cc:1918] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:27.087419 660 log.cc:1196] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/596932cfa4ac4a62afcac553094fe057
I20260927 07:06:27.087677 659 ts_tablet_manager.cc:1939] T 596932cfa4ac4a62afcac553094fe057 P bb54f3198c084ad29daaa89edbfcb954: Deleting consensus metadata
I20260927 07:06:27.087862 661 tablet_replica.cc:330] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: stopping tablet replica
I20260927 07:06:27.088078 661 raft_consensus.cc:2229] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:27.088120 342 tablet_service.cc:1467] Processing CreateTablet for tablet de6f6159033a4c3a8e209358873df917 (DEFAULT_TABLE table=test-table [id=2c66709fd12d432fb2574a277fa00a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:27.088429 661 raft_consensus.cc:2258] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:27.089385 32520 catalog_manager.cc:4841] TS bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561): tablet 596932cfa4ac4a62afcac553094fe057 (table test-table [id=24f45903c903455cb355d3e1bae25e8a]) successfully deleted
I20260927 07:06:27.089368 486 tablet_service.cc:1467] Processing CreateTablet for tablet de6f6159033a4c3a8e209358873df917 (DEFAULT_TABLE table=test-table [id=2c66709fd12d432fb2574a277fa00a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:27.088577 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6f6159033a4c3a8e209358873df917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:27.089555 32675 tablet_service.cc:1467] Processing CreateTablet for tablet de6f6159033a4c3a8e209358873df917 (DEFAULT_TABLE table=test-table [id=2c66709fd12d432fb2574a277fa00a46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:27.089763 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6f6159033a4c3a8e209358873df917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:27.089931 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6f6159033a4c3a8e209358873df917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:27.089588 661 ts_tablet_manager.cc:1905] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:27.096062 671 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Bootstrap starting.
I20260927 07:06:27.096096 672 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Bootstrap starting.
I20260927 07:06:27.097571 671 tablet_bootstrap.cc:654] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:27.097652 672 tablet_bootstrap.cc:654] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:27.101099 661 ts_tablet_manager.cc:1918] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:06:27.101287 661 log.cc:1196] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/596932cfa4ac4a62afcac553094fe057
I20260927 07:06:27.101718 661 ts_tablet_manager.cc:1939] T 596932cfa4ac4a62afcac553094fe057 P fd3b5057b1f0416a9febd1c937226e5a: Deleting consensus metadata
I20260927 07:06:27.102913 32520 catalog_manager.cc:4841] TS fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399): tablet 596932cfa4ac4a62afcac553094fe057 (table test-table [id=24f45903c903455cb355d3e1bae25e8a]) successfully deleted
I20260927 07:06:27.103657 660 ts_tablet_manager.cc:1939] T 596932cfa4ac4a62afcac553094fe057 P 3960695285d843fa957588855353b1bc: Deleting consensus metadata
I20260927 07:06:27.105011 670 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Bootstrap starting.
I20260927 07:06:27.105015 32521 catalog_manager.cc:4841] TS 3960695285d843fa957588855353b1bc (127.30.16.67:34405): tablet 596932cfa4ac4a62afcac553094fe057 (table test-table [id=24f45903c903455cb355d3e1bae25e8a]) successfully deleted
I20260927 07:06:27.106276 671 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: No bootstrap required, opened a new log
I20260927 07:06:27.106470 671 ts_tablet_manager.cc:1397] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20260927 07:06:27.107015 670 tablet_bootstrap.cc:654] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:27.107440 671 raft_consensus.cc:348] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.107754 671 raft_consensus.cc:374] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:27.107844 671 raft_consensus.cc:729] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb54f3198c084ad29daaa89edbfcb954, State: Initialized, Role: FOLLOWER
I20260927 07:06:27.108098 671 consensus_queue.cc:260] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.108691 671 ts_tablet_manager.cc:1428] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 07:06:27.109721 670 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: No bootstrap required, opened a new log
I20260927 07:06:27.109923 670 ts_tablet_manager.cc:1397] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260927 07:06:27.110702 670 raft_consensus.cc:348] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.110951 670 raft_consensus.cc:374] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:27.111027 670 raft_consensus.cc:729] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Initialized, Role: FOLLOWER
I20260927 07:06:27.111304 670 consensus_queue.cc:260] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.111922 670 ts_tablet_manager.cc:1428] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:06:27.113549 672 tablet_bootstrap.cc:492] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: No bootstrap required, opened a new log
I20260927 07:06:27.113775 672 ts_tablet_manager.cc:1397] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20260927 07:06:27.114600 672 raft_consensus.cc:348] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.114841 672 raft_consensus.cc:374] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:27.114925 672 raft_consensus.cc:729] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960695285d843fa957588855353b1bc, State: Initialized, Role: FOLLOWER
I20260927 07:06:27.115159 672 consensus_queue.cc:260] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.115734 672 ts_tablet_manager.cc:1428] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 07:06:27.406018 638 raft_consensus.cc:482] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:27.406224 638 raft_consensus.cc:504] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.407020 638 leader_election.cc:288] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:27.407737 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6f6159033a4c3a8e209358873df917" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc" is_pre_election: true
I20260927 07:06:27.407737 32695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6f6159033a4c3a8e209358873df917" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954" is_pre_election: true
I20260927 07:06:27.408006 506 raft_consensus.cc:2454] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:27.408035 32695 raft_consensus.cc:2454] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 0.
I20260927 07:06:27.408558 32764 leader_election.cc:302] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:27.408859 638 raft_consensus.cc:2781] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:27.409017 638 raft_consensus.cc:482] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:27.409126 638 raft_consensus.cc:3037] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:27.411168 638 raft_consensus.cc:504] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.411739 638 leader_election.cc:288] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Requested vote from peers bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561), 3960695285d843fa957588855353b1bc (127.30.16.67:34405)
I20260927 07:06:27.412308 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6f6159033a4c3a8e209358873df917" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960695285d843fa957588855353b1bc"
I20260927 07:06:27.412343 32695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6f6159033a4c3a8e209358873df917" candidate_uuid: "fd3b5057b1f0416a9febd1c937226e5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb54f3198c084ad29daaa89edbfcb954"
I20260927 07:06:27.412503 506 raft_consensus.cc:3037] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:27.412551 32695 raft_consensus.cc:3037] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:27.414634 506 raft_consensus.cc:2454] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:27.414677 32695 raft_consensus.cc:2454] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd3b5057b1f0416a9febd1c937226e5a in term 1.
I20260927 07:06:27.415129 32764 leader_election.cc:302] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960695285d843fa957588855353b1bc, fd3b5057b1f0416a9febd1c937226e5a; no voters:
I20260927 07:06:27.415411 638 raft_consensus.cc:2781] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:27.415575 638 raft_consensus.cc:686] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Becoming Leader. State: Replica: fd3b5057b1f0416a9febd1c937226e5a, State: Running, Role: LEADER
I20260927 07:06:27.415884 638 consensus_queue.cc:237] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } }
I20260927 07:06:27.418701 32533 catalog_manager.cc:5495] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a reported cstate change: term changed from 0 to 1, leader changed from <none> to fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66). New cstate: current_term: 1 leader_uuid: "fd3b5057b1f0416a9febd1c937226e5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd3b5057b1f0416a9febd1c937226e5a" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:27.481454 32695 raft_consensus.cc:1261] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:27.481434 506 raft_consensus.cc:1261] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Refusing update from remote peer fd3b5057b1f0416a9febd1c937226e5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:27.481992 678 consensus_queue.cc:1035] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960695285d843fa957588855353b1bc" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:27.482445 638 consensus_queue.cc:1035] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb54f3198c084ad29daaa89edbfcb954" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 07:06:27.501938 636 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 07:06:29.089761 32533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:29.089975 32533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:29.094007 32533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 929bf5157abb4b5c921b4638bc16a2f0: Sending DeleteTablet for 3 replicas of tablet de6f6159033a4c3a8e209358873df917
I20260927 07:06:29.094944 32675 tablet_service.cc:1514] Processing DeleteTablet for tablet de6f6159033a4c3a8e209358873df917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:29 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37380
I20260927 07:06:29.094969 486 tablet_service.cc:1514] Processing DeleteTablet for tablet de6f6159033a4c3a8e209358873df917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:29 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55366
I20260927 07:06:29.095003 342 tablet_service.cc:1514] Processing DeleteTablet for tablet de6f6159033a4c3a8e209358873df917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:29 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47208
I20260927 07:06:29.095693 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32605
I20260927 07:06:29.097767 711 tablet_replica.cc:330] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: stopping tablet replica
I20260927 07:06:29.098040 711 raft_consensus.cc:2229] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:29.098312 711 raft_consensus.cc:2258] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:29.098879 713 tablet_replica.cc:330] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: stopping tablet replica
I20260927 07:06:29.099130 713 raft_consensus.cc:2229] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:29.099406 713 raft_consensus.cc:2258] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:29.100567 712 tablet_replica.cc:330] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: stopping tablet replica
I20260927 07:06:29.100829 712 raft_consensus.cc:2229] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:29.101312 712 raft_consensus.cc:2258] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:29.102461 711 ts_tablet_manager.cc:1905] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:29.102856 713 ts_tablet_manager.cc:1905] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:29.103307 712 ts_tablet_manager.cc:1905] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:29.109365 713 ts_tablet_manager.cc:1918] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:29.109505 713 log.cc:1196] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/de6f6159033a4c3a8e209358873df917
I20260927 07:06:29.110014 713 ts_tablet_manager.cc:1939] T de6f6159033a4c3a8e209358873df917 P 3960695285d843fa957588855353b1bc: Deleting consensus metadata
I20260927 07:06:29.110015 711 ts_tablet_manager.cc:1918] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:29.110143 711 log.cc:1196] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/de6f6159033a4c3a8e209358873df917
I20260927 07:06:29.110610 711 ts_tablet_manager.cc:1939] T de6f6159033a4c3a8e209358873df917 P bb54f3198c084ad29daaa89edbfcb954: Deleting consensus metadata
I20260927 07:06:29.111258 32521 catalog_manager.cc:4841] TS 3960695285d843fa957588855353b1bc (127.30.16.67:34405): tablet de6f6159033a4c3a8e209358873df917 (table test-table [id=2c66709fd12d432fb2574a277fa00a46]) successfully deleted
I20260927 07:06:29.111351 712 ts_tablet_manager.cc:1918] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 07:06:29.111605 712 log.cc:1196] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/de6f6159033a4c3a8e209358873df917
I20260927 07:06:29.111672 32520 catalog_manager.cc:4841] TS bb54f3198c084ad29daaa89edbfcb954 (127.30.16.65:36561): tablet de6f6159033a4c3a8e209358873df917 (table test-table [id=2c66709fd12d432fb2574a277fa00a46]) successfully deleted
I20260927 07:06:29.112107 712 ts_tablet_manager.cc:1939] T de6f6159033a4c3a8e209358873df917 P fd3b5057b1f0416a9febd1c937226e5a: Deleting consensus metadata
I20260927 07:06:29.113085 32520 catalog_manager.cc:4841] TS fd3b5057b1f0416a9febd1c937226e5a (127.30.16.66:46399): tablet de6f6159033a4c3a8e209358873df917 (table test-table [id=2c66709fd12d432fb2574a277fa00a46]) successfully deleted
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:29.123661 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:29.097091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37396 (local address 127.30.16.65:36561)
0927 07:06:29.108631 (+ 11540us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:29.108637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:29.108663 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:29.108723 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:29.108728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:29.108814 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:29.108933 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:29.110858 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.112148 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.113749 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.114170 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.114268 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:29.118139 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:29.118154 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:29.118172 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:29.118205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:29.121029 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.121707 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.121714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.121717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.121850 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.122188 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.122191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.122193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.122618 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:29.122873 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:29.123200 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 07:06:29.123364 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11403,"thread_start_us":13595,"threads_started":1}
W20260927 07:06:29.124353 32736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37396
W20260927 07:06:29.124957 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:29.125187 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 32605
I20260927 07:06:29.143246 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32745
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:29.159956 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:29.144335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.30.16.66:46399)
0927 07:06:29.144897 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:29.144902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:29.144918 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:06:29.144973 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:29.144977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:29.145043 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:29.145174 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:29.146637 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.147451 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.148997 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.149430 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.149563 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:29.153841 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:29.153856 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:29.153872 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:29.153916 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:29.157046 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.157770 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.157777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.157780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.157931 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.158307 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.158313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.158317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.158746 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:29.159057 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:29.159371 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 07:06:29.159600 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
W20260927 07:06:29.160635 407 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47248
W20260927 07:06:29.161266 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32745: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:29.161409 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 32745
I20260927 07:06:29.188500 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 416
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[32489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:29.205143 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:29.189355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.30.16.67:34405)
0927 07:06:29.189893 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:29.189898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:29.189914 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:06:29.189968 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:29.189972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:29.190037 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:29.190131 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:29.191591 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.192482 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.193782 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:29.194259 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:29.194398 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:29.199131 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:29.199148 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:29.199164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:29.199209 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:29.202219 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.203290 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.203298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.203301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.203438 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:29.203784 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:29.203787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:29.203789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:29.204147 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:29.204377 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:29.204703 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 07:06:29.204872 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
W20260927 07:06:29.205894 546 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55384
W20260927 07:06:29.206533 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 416: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:29.206799 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 416
I20260927 07:06:29.229607 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 32502
I20260927 07:06:29.230945 32574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:29.348910 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 32502
2026-09-27T07:06:29Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8461 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[721](info): set up 2 sockets
Sep 27 07:06:29 dist-test-slave-8vqf krb5kdc[721](info): commencing operation
krb5kdc: starting...
W20260927 07:06:30.530841 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:30 dist-test-slave-8vqf krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:30.724313 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33109
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:31.048501 734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:31.048882 734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:31.049062 734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:31.057265 734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:31.057446 734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:31.057560 734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:31.057648 734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:31.057739 734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:31.068104 734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:31.069653 734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:31.071134 734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:31.083458 742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:31.083575 739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:31.083513 740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:31.084311 734 server_base.cc:1034] running on GCE node
I20260927 07:06:31.085060 734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:31.086611 734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:31.087828 734 hybrid_clock.cc:648] HybridClock initialized: now 1790492791087783 us; error 38 us; skew 500 ppm
I20260927 07:06:31.088171 734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:31 dist-test-slave-8vqf krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:31.091323 734 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:31.092921 734 webserver.cc:460] Webserver started at http://127.30.16.126:45957/ using document root <none> and password file <none>
I20260927 07:06:31.093686 734 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:31.093811 734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:31.094144 734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:31.097245 734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "51877a7d14ed471aa1fcc76735a32cb4"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "e552c1f75cba54d4e206c49a2c7a1602"
server_key_iv: "95d027d15af7314e44d645f600e11f5e"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.098038 734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "51877a7d14ed471aa1fcc76735a32cb4"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "e552c1f75cba54d4e206c49a2c7a1602"
server_key_iv: "95d027d15af7314e44d645f600e11f5e"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.103398 734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 07:06:31.107030 749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:31.108343 734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:31.108549 734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "51877a7d14ed471aa1fcc76735a32cb4"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "e552c1f75cba54d4e206c49a2c7a1602"
server_key_iv: "95d027d15af7314e44d645f600e11f5e"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.108743 734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:31.120069 734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:31.124713 734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:31.125023 734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:31.143877 734 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33109
I20260927 07:06:31.143927 800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33109 every 8 connection(s)
I20260927 07:06:31.145318 734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:31.149408 801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:31.152959 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 734
I20260927 07:06:31.153247 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:31.154110 30785 external_mini_cluster.cc:1428] Setting key cf78ebdd76907efec82ceeb006503c28
I20260927 07:06:31.164768 801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4: Bootstrap starting.
I20260927 07:06:31.168608 801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:31.169754 801 log.cc:824] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:06:31 dist-test-slave-8vqf krb5kdc[721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:31.173193 801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4: No bootstrap required, opened a new log
I20260927 07:06:31.180861 804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:31.156399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.30.16.126:33109)
0927 07:06:31.157213 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:31.157230 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:31.157287 (+ 57us) server_negotiation.cc:407] Connection header received
0927 07:06:31.159880 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:31.159914 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:31.160527 (+ 613us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:31.161045 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:31.162317 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.164515 (+ 2198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.165859 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.167140 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.167333 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:31.171321 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:31.171329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:31.171345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:31.171386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:31.175724 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:31.176762 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:31.176772 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:31.176781 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:31.177026 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:31.177443 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:31.177448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:31.177452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:31.178190 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:31.178465 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:31.178953 (+ 488us) server_negotiation.cc:299] Negotiation successful
0927 07:06:31.179432 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260927 07:06:31.184181 801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } }
I20260927 07:06:31.184520 801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:31.184600 801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51877a7d14ed471aa1fcc76735a32cb4, State: Initialized, Role: FOLLOWER
I20260927 07:06:31.185072 801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } }
I20260927 07:06:31.185268 801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:31.185393 801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:31.185515 801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:31.187779 801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } }
I20260927 07:06:31.188223 801 leader_election.cc:302] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51877a7d14ed471aa1fcc76735a32cb4; no voters:
I20260927 07:06:31.188858 801 leader_election.cc:288] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:31.189087 806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:31.190949 806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 51877a7d14ed471aa1fcc76735a32cb4, State: Running, Role: LEADER
I20260927 07:06:31.191138 801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:31.191524 806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } }
I20260927 07:06:31.194906 808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51877a7d14ed471aa1fcc76735a32cb4. Latest consensus state: current_term: 1 leader_uuid: "51877a7d14ed471aa1fcc76735a32cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } } }
I20260927 07:06:31.195253 808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:31.195822 807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51877a7d14ed471aa1fcc76735a32cb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51877a7d14ed471aa1fcc76735a32cb4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33109 } } }
I20260927 07:06:31.196132 807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:31.196401 816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:31.201234 816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:31.209851 816 catalog_manager.cc:1347] Generated new cluster ID: 241c46e28f874fdca4d7df8063d92c23
I20260927 07:06:31.209973 816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:31.229509 816 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:31.231529 816 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:31.240031 816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51877a7d14ed471aa1fcc76735a32cb4: Generated new TSK 0
I20260927 07:06:31.241153 816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:31.291594 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:31.624429 827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:31.624771 827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:31.624951 827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:31.632448 827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:31.632609 827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:31.632797 827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:31.642800 827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:31.644155 827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:31.645502 827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:31.658805 832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:31.659036 833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:31.658856 835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:31.659737 827 server_base.cc:1034] running on GCE node
I20260927 07:06:31.660351 827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:31.661440 827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:31.662662 827 hybrid_clock.cc:648] HybridClock initialized: now 1790492791662599 us; error 56 us; skew 500 ppm
I20260927 07:06:31.663002 827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:31 dist-test-slave-8vqf krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:31.666226 827 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:31.667833 827 webserver.cc:460] Webserver started at http://127.30.16.65:34507/ using document root <none> and password file <none>
I20260927 07:06:31.668502 827 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:31.668598 827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:31.668852 827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:31.671914 827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "5bac0844a1d04d03ae1cfe29aa08e1bc"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "f0b24c6af53467427867799dea9bde6b"
server_key_iv: "700ce2842bcf11b78332e97570ba72f5"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.672648 827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "5bac0844a1d04d03ae1cfe29aa08e1bc"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "f0b24c6af53467427867799dea9bde6b"
server_key_iv: "700ce2842bcf11b78332e97570ba72f5"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.677744 827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 07:06:31.681203 842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:31.682617 827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:06:31.682796 827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "5bac0844a1d04d03ae1cfe29aa08e1bc"
format_stamp: "Formatted at 2026-09-27 07:06:31 on dist-test-slave-8vqf"
server_key: "f0b24c6af53467427867799dea9bde6b"
server_key_iv: "700ce2842bcf11b78332e97570ba72f5"
server_key_version: "encryptionkey@0"
I20260927 07:06:31.683012 827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:31.694940 827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:31.699348 827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:31.699594 827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:31.700476 827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:31.701889 827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:31.701959 827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:31.702042 827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:31.702080 827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:31.734812 827 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45469
I20260927 07:06:31.734858 954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45469 every 8 connection(s)
I20260927 07:06:31.736172 827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:31.742513 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 827
I20260927 07:06:31.742805 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:31.743602 30785 external_mini_cluster.cc:1428] Setting key da986640df1e4d68524d53b7c0b1f441
Sep 27 07:06:31 dist-test-slave-8vqf krb5kdc[721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492791, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:31.759145 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:31.738705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60597 (local address 127.30.16.126:33109)
0927 07:06:31.739187 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:31.739191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:31.740214 (+ 1023us) server_negotiation.cc:407] Connection header received
0927 07:06:31.742086 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:31.742091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:31.742156 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:31.742255 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:31.744423 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.745447 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.747952 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.748336 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.748436 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:31.753026 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:31.753031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:31.753034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:31.753065 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:31.755671 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:31.756642 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:31.756646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:31.756648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:31.756755 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:31.757248 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:31.757251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:31.757253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:31.757496 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:31.757597 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:31.758623 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 07:06:31.758789 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":292,"threads_started":1}
I20260927 07:06:31.760761 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:31.739341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33109 (local address 127.30.16.65:60597)
0927 07:06:31.740049 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:31.740087 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:31.741797 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:31.742608 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:31.742621 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:31.743412 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:31.744244 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:31.744265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.745590 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.745594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:31.747789 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:31.747812 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:31.748576 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:31.748585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:31.748914 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:31.749648 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:31.749684 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:31.752772 (+ 3088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:31.755820 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:31.755832 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:31.755854 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:31.756520 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:31.756857 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:31.756866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:31.756873 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:31.757135 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:31.757625 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:31.757636 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:31.758137 (+ 501us) client_negotiation.cc:769] Sending connection context
0927 07:06:31.758468 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 07:06:31.759128 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":258,"threads_started":1}
I20260927 07:06:31.762452 955 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33109
I20260927 07:06:31.762820 955 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:31.763579 955 heartbeater.cc:511] Master 127.30.16.126:33109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:06:31.765952 766 ts_manager.cc:194] Registered new tserver with Master: 5bac0844a1d04d03ae1cfe29aa08e1bc (127.30.16.65:45469)
I20260927 07:06:31.767478 766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:60597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:31.790916 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:32.128275 961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:32.128588 961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:32.128751 961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:32.136505 961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:32.136641 961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:32.136821 961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:32.149685 961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:32.151346 961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:32.153028 961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:32.168517 966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:32.168565 969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:32.168524 967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:32.169705 961 server_base.cc:1034] running on GCE node
I20260927 07:06:32.170351 961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:32.171397 961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:32.172657 961 hybrid_clock.cc:648] HybridClock initialized: now 1790492792172593 us; error 53 us; skew 500 ppm
I20260927 07:06:32.173072 961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:32 dist-test-slave-8vqf krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492792, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:32.183833 961 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:32.185470 961 webserver.cc:460] Webserver started at http://127.30.16.66:39335/ using document root <none> and password file <none>
I20260927 07:06:32.186245 961 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:32.186399 961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:32.186836 961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:32.190142 961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "8173a65106f2461fa23c6f6f5754511d"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "3ac31c8bd654953b359d3e27dfac80bd"
server_key_iv: "293bffdf4c68960c8a00ccdf22999384"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.191016 961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "8173a65106f2461fa23c6f6f5754511d"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "3ac31c8bd654953b359d3e27dfac80bd"
server_key_iv: "293bffdf4c68960c8a00ccdf22999384"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.196843 961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:32.200680 976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.202153 961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:06:32.202353 961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "8173a65106f2461fa23c6f6f5754511d"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "3ac31c8bd654953b359d3e27dfac80bd"
server_key_iv: "293bffdf4c68960c8a00ccdf22999384"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.202628 961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:32.220748 961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:32.225266 961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:32.225576 961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:32.226620 961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:32.227998 961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:32.228098 961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.228251 961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:32.228325 961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.262036 961 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:35283
I20260927 07:06:32.262090 1088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:35283 every 8 connection(s)
I20260927 07:06:32.263443 961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:32.271821 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 961
I20260927 07:06:32.272034 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:32.272462 30785 external_mini_cluster.cc:1428] Setting key 10e936a1fc7ebf111fb7140df586aa97
Sep 27 07:06:32 dist-test-slave-8vqf krb5kdc[721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492792, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:32.285034 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:32.266051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:41513 (local address 127.30.16.126:33109)
0927 07:06:32.266500 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:32.266504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:32.267464 (+ 960us) server_negotiation.cc:407] Connection header received
0927 07:06:32.269267 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:32.269273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:32.269365 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:32.269486 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:32.271061 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.271767 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.273463 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.273916 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.274046 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:32.277874 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:32.277884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:32.277892 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:32.277941 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:32.281124 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:32.282117 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:32.282122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:32.282125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:32.282254 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:32.282955 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:32.282964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:32.282970 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:32.283388 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:32.283544 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:32.284564 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0927 07:06:32.284736 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":252,"threads_started":1}
I20260927 07:06:32.287556 1092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:32.266613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33109 (local address 127.30.16.66:41513)
0927 07:06:32.267297 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:32.267340 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:32.268995 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:32.269738 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:32.269746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:32.270245 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:32.270895 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:32.270912 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.271917 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.271923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:32.273310 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:32.273321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.274146 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.274152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:32.274400 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:32.275143 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:32.275170 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:32.277592 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:32.281264 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:32.281276 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:32.281298 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:32.281961 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:32.282365 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:32.282371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:32.282376 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:32.282696 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:32.283535 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:32.283547 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:32.284086 (+ 539us) client_negotiation.cc:769] Sending connection context
0927 07:06:32.284428 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 07:06:32.285451 (+ 1023us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":263,"threads_started":1}
I20260927 07:06:32.289458 1089 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33109
I20260927 07:06:32.289791 1089 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:32.290639 1089 heartbeater.cc:511] Master 127.30.16.126:33109 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:32.292264 766 ts_manager.cc:194] Registered new tserver with Master: 8173a65106f2461fa23c6f6f5754511d (127.30.16.66:35283)
I20260927 07:06:32.293304 766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:41513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:32.319725 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:36769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:32.652717 1095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:32.653024 1095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:32.653174 1095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:32.660717 1095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:32.660845 1095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:32.661026 1095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:32.671157 1095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:32.672544 1095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:32.673906 1095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:32.687583 1100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:32.687619 1101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:32.687543 1103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:32.688783 1095 server_base.cc:1034] running on GCE node
I20260927 07:06:32.689519 1095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:32.690723 1095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:32.692047 1095 hybrid_clock.cc:648] HybridClock initialized: now 1790492792691991 us; error 68 us; skew 500 ppm
I20260927 07:06:32.692381 1095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:32 dist-test-slave-8vqf krb5kdc[721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492792, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:32.695802 1095 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:32.697434 1095 webserver.cc:460] Webserver started at http://127.30.16.67:40971/ using document root <none> and password file <none>
I20260927 07:06:32.698151 1095 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:32.698282 1095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:32.698719 1095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:32.701709 1095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "5953732298774628aec353c20fafb911"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "e61b02de3dd9635ea519edc436ac5b77"
server_key_iv: "9249d3e7b808a1306d53f6b1863bb7bf"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.702647 1095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "5953732298774628aec353c20fafb911"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "e61b02de3dd9635ea519edc436ac5b77"
server_key_iv: "9249d3e7b808a1306d53f6b1863bb7bf"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.708138 1095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 07:06:32.711714 1110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.713318 1095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:32.713508 1095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "5953732298774628aec353c20fafb911"
format_stamp: "Formatted at 2026-09-27 07:06:32 on dist-test-slave-8vqf"
server_key: "e61b02de3dd9635ea519edc436ac5b77"
server_key_iv: "9249d3e7b808a1306d53f6b1863bb7bf"
server_key_version: "encryptionkey@0"
I20260927 07:06:32.713716 1095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:32.746481 1095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:32.751076 1095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:32.751390 1095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:32.752401 1095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:32.753757 1095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:32.753837 1095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.754006 1095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:32.754058 1095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:32.770360 955 heartbeater.cc:503] Master 127.30.16.126:33109 was elected leader, sending a full tablet report...
I20260927 07:06:32.788276 1095 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:46239
I20260927 07:06:32.788368 1222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:46239 every 8 connection(s)
I20260927 07:06:32.789649 1095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:32.799293 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1095
I20260927 07:06:32.799505 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:32.799948 30785 external_mini_cluster.cc:1428] Setting key cc3128f417f349748f33c7ee1c86715d
Sep 27 07:06:32 dist-test-slave-8vqf krb5kdc[721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492792, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:32.810778 1225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:32.792441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:56745 (local address 127.30.16.126:33109)
0927 07:06:32.792902 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:32.792908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:32.793988 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 07:06:32.796084 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:32.796088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:32.796174 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:32.796278 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:32.797863 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.798750 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.800634 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.801113 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.801272 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:32.805123 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:32.805127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:32.805131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:32.805164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:32.807910 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:32.808618 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:32.808621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:32.808624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:32.808734 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:32.809122 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:32.809125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:32.809127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:32.809406 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:32.809522 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:32.810252 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 07:06:32.810421 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":211,"threads_started":1}
I20260927 07:06:32.811857 1226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:32.793061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33109 (local address 127.30.16.67:56745)
0927 07:06:32.793825 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:32.793864 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:32.795731 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:32.796498 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:32.796506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:32.797072 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:32.797662 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:32.797685 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.798968 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.798972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:32.800470 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:32.800485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:32.801335 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:32.801341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:32.801676 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:32.802446 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:32.802475 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:32.804885 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:32.808044 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:32.808052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:32.808066 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:32.808520 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:32.808826 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:32.808829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:32.808832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:32.809028 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:32.809551 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:32.809558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:32.809909 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 07:06:32.810164 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 07:06:32.810670 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":277,"threads_started":1}
I20260927 07:06:32.813387 1223 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33109
I20260927 07:06:32.813795 1223 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:32.814747 1223 heartbeater.cc:511] Master 127.30.16.126:33109 requested a full tablet report, sending...
I20260927 07:06:32.816296 766 ts_manager.cc:194] Registered new tserver with Master: 5953732298774628aec353c20fafb911 (127.30.16.67:46239)
I20260927 07:06:32.817075 766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:56745
I20260927 07:06:32.828428 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:06:32.859191 1225 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:32.856578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.30.16.126:33109)
0927 07:06:32.856911 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:32.856919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:32.857179 (+ 260us) server_negotiation.cc:407] Connection header received
0927 07:06:32.857420 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:32.857424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:32.857489 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:32.857600 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:32.858979 (+ 1379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53674: BlockingRecv error: recv got EOF from 127.0.0.1:53674 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 07:06:32.862406 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 827
W20260927 07:06:32.865440 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 827: Not authorized: Client connection negotiation failed: client connection to 127.30.16.65:45469: server requires authentication, but client does not have Kerberos credentials available
I20260927 07:06:32.865589 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 827
I20260927 07:06:32.881524 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 961
W20260927 07:06:32.884421 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 961: Not authorized: Client connection negotiation failed: client connection to 127.30.16.66:35283: server requires authentication, but client does not have Kerberos credentials available
I20260927 07:06:32.884614 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 961
I20260927 07:06:32.901329 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1095
W20260927 07:06:32.904423 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1095: Not authorized: Client connection negotiation failed: client connection to 127.30.16.67:46239: server requires authentication, but client does not have Kerberos credentials available
I20260927 07:06:32.904580 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1095
I20260927 07:06:32.920185 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 734
I20260927 07:06:32.921437 796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:33.024209 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 734
2026-09-27T07:06:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3673 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:33 dist-test-slave-8vqf krb5kdc[1241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:33 dist-test-slave-8vqf krb5kdc[1241](info): set up 2 sockets
Sep 27 07:06:33 dist-test-slave-8vqf krb5kdc[1241](info): commencing operation
krb5kdc: starting...
W20260927 07:06:34.135039 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:34 dist-test-slave-8vqf krb5kdc[1241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:34.334656 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40881
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:39401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:40881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:34.676811 1254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:34.677183 1254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:34.677361 1254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:34.684708 1254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:34.684876 1254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:34.685009 1254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:34.685107 1254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:34.685199 1254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:34.695245 1254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:40881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:34.696633 1254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:34.697983 1254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:34.710474 1260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:34.710501 1259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:34.710521 1262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:34.711407 1254 server_base.cc:1034] running on GCE node
I20260927 07:06:34.712281 1254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:34.713843 1254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:34.715091 1254 hybrid_clock.cc:648] HybridClock initialized: now 1790492794715040 us; error 47 us; skew 500 ppm
I20260927 07:06:34.715459 1254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:34 dist-test-slave-8vqf krb5kdc[1241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:34.718813 1254 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:34.720224 1254 webserver.cc:460] Webserver started at http://127.30.16.126:41719/ using document root <none> and password file <none>
I20260927 07:06:34.720884 1254 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:34.720984 1254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:34.721256 1254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:34.724409 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "d741b6a4e8ec491cb3a6f04a63979c5b"
format_stamp: "Formatted at 2026-09-27 07:06:34 on dist-test-slave-8vqf"
server_key: "ad43f8a1ee925b7433308188095f6d9d"
server_key_iv: "c6783183de7dd13ecf4c955224abfd7d"
server_key_version: "encryptionkey@0"
I20260927 07:06:34.725155 1254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "d741b6a4e8ec491cb3a6f04a63979c5b"
format_stamp: "Formatted at 2026-09-27 07:06:34 on dist-test-slave-8vqf"
server_key: "ad43f8a1ee925b7433308188095f6d9d"
server_key_iv: "c6783183de7dd13ecf4c955224abfd7d"
server_key_version: "encryptionkey@0"
I20260927 07:06:34.730400 1254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 07:06:34.734016 1269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:34.735543 1254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:34.735682 1254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "d741b6a4e8ec491cb3a6f04a63979c5b"
format_stamp: "Formatted at 2026-09-27 07:06:34 on dist-test-slave-8vqf"
server_key: "ad43f8a1ee925b7433308188095f6d9d"
server_key_iv: "c6783183de7dd13ecf4c955224abfd7d"
server_key_version: "encryptionkey@0"
I20260927 07:06:34.735852 1254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:34.753307 1254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:34.757495 1254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:34.757727 1254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:34.776190 1254 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:40881
I20260927 07:06:34.776225 1320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:40881 every 8 connection(s)
I20260927 07:06:34.777601 1254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:34.781525 1321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:34.783082 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1254
I20260927 07:06:34.783288 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:34.784097 30785 external_mini_cluster.cc:1428] Setting key 8769d28bc4b8715e191aaba2237547b7
I20260927 07:06:34.796413 1321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b: Bootstrap starting.
Sep 27 07:06:34 dist-test-slave-8vqf krb5kdc[1241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:34.800192 1321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:34.801453 1321 log.cc:824] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:34.804400 1321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b: No bootstrap required, opened a new log
I20260927 07:06:34.807492 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:34.785984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45088 (local address 127.30.16.126:40881)
0927 07:06:34.786845 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:34.786860 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:34.786903 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:06:34.788471 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:34.788496 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:34.788910 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:34.789244 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:34.791145 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:34.792404 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:34.794294 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:34.795191 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:34.795346 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:34.800003 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:34.800011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:34.800026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:34.800067 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:34.803274 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:34.804003 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:34.804009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:34.804016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:34.804171 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:34.804631 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:34.804634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:34.804636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:34.805188 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:34.805492 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:34.805940 (+ 448us) server_negotiation.cc:299] Negotiation successful
0927 07:06:34.806389 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":417,"threads_started":1}
I20260927 07:06:34.811474 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:06:34.812546 1321 raft_consensus.cc:348] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } }
I20260927 07:06:34.812937 1321 raft_consensus.cc:374] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:34.813040 1321 raft_consensus.cc:729] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d741b6a4e8ec491cb3a6f04a63979c5b, State: Initialized, Role: FOLLOWER
I20260927 07:06:34.813598 1321 consensus_queue.cc:260] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } }
I20260927 07:06:34.813834 1321 raft_consensus.cc:388] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:34.813987 1321 raft_consensus.cc:482] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:34.814160 1321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:34.816380 1321 raft_consensus.cc:504] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } }
I20260927 07:06:34.816874 1321 leader_election.cc:302] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d741b6a4e8ec491cb3a6f04a63979c5b; no voters:
I20260927 07:06:34.817552 1321 leader_election.cc:288] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:34.817665 1327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:34.819228 1327 raft_consensus.cc:686] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [term 1 LEADER]: Becoming Leader. State: Replica: d741b6a4e8ec491cb3a6f04a63979c5b, State: Running, Role: LEADER
I20260927 07:06:34.819761 1327 consensus_queue.cc:237] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } }
I20260927 07:06:34.819943 1321 sys_catalog.cc:564] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:34.822995 1328 sys_catalog.cc:455] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } } }
I20260927 07:06:34.823436 1328 sys_catalog.cc:458] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:34.822597 1329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d741b6a4e8ec491cb3a6f04a63979c5b. Latest consensus state: current_term: 1 leader_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d741b6a4e8ec491cb3a6f04a63979c5b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40881 } } }
I20260927 07:06:34.825771 1329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:34.827622 1335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:34.833338 1335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:34.848070 1335 catalog_manager.cc:1347] Generated new cluster ID: 74457860850f49dd885449b467ac9a3a
I20260927 07:06:34.848191 1335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:34.872363 1335 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:34.874835 1335 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:34.888088 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:34.873061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.30.16.126:40881)
0927 07:06:34.873251 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:34.873258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:34.879385 (+ 6127us) server_negotiation.cc:407] Connection header received
0927 07:06:34.879584 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:34.879590 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:34.879674 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:34.879795 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:34.881166 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:34.882277 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:34.884295 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:34.884956 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:34.885088 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:34.886384 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:34.886391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:34.886395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:34.887245 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:34.887314 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:34.887342 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:34.887486 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:34.887630 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:06:34.887734 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:06:34.889293 1335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d741b6a4e8ec491cb3a6f04a63979c5b: Generated new TSK 0
I20260927 07:06:34.890484 1335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:06:34.900476 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1254
I20260927 07:06:34.901647 1316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:34.992425 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1254
2026-09-27T07:06:35Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1961 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:35 dist-test-slave-8vqf krb5kdc[1355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:35 dist-test-slave-8vqf krb5kdc[1355](info): set up 2 sockets
Sep 27 07:06:35 dist-test-slave-8vqf krb5kdc[1355](info): commencing operation
krb5kdc: starting...
W20260927 07:06:36.079213 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.007s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:36 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:36.282084 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42729
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:42729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:36.627070 1368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:36.627425 1368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:36.627597 1368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:36.634997 1368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:36.635143 1368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:36.635258 1368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:36.635362 1368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:36.635447 1368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:36.645653 1368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:42729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:36.647167 1368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:36.648499 1368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:36.660554 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:36.660528 1376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:36.660490 1374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:36.661418 1368 server_base.cc:1034] running on GCE node
I20260927 07:06:36.662256 1368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:36.663921 1368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:36.665165 1368 hybrid_clock.cc:648] HybridClock initialized: now 1790492796665112 us; error 46 us; skew 500 ppm
I20260927 07:06:36.665537 1368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:36 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492796, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:36.668941 1368 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:36.670725 1368 webserver.cc:460] Webserver started at http://127.30.16.126:40645/ using document root <none> and password file <none>
I20260927 07:06:36.671505 1368 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:36.671669 1368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:36.672083 1368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:36.675589 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "46019cae98c546e5b8581a4bd55ae49e"
format_stamp: "Formatted at 2026-09-27 07:06:36 on dist-test-slave-8vqf"
server_key: "ed76f5754c2a78aa1c135888d17e5315"
server_key_iv: "a91c34526aca46e385b99912e2d893d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:36.676424 1368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "46019cae98c546e5b8581a4bd55ae49e"
format_stamp: "Formatted at 2026-09-27 07:06:36 on dist-test-slave-8vqf"
server_key: "ed76f5754c2a78aa1c135888d17e5315"
server_key_iv: "a91c34526aca46e385b99912e2d893d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:36.682137 1368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:36.686043 1383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:36.687577 1368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:36.687772 1368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "46019cae98c546e5b8581a4bd55ae49e"
format_stamp: "Formatted at 2026-09-27 07:06:36 on dist-test-slave-8vqf"
server_key: "ed76f5754c2a78aa1c135888d17e5315"
server_key_iv: "a91c34526aca46e385b99912e2d893d0"
server_key_version: "encryptionkey@0"
I20260927 07:06:36.687975 1368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:36.701522 1368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:36.705647 1368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:36.705953 1368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:36.724830 1368 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:42729
I20260927 07:06:36.724838 1434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:42729 every 8 connection(s)
I20260927 07:06:36.726248 1368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:36.730440 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:36.731570 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1368
I20260927 07:06:36.731789 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:36.732654 30785 external_mini_cluster.cc:1428] Setting key c75cdf5f66005280363972a2fb54793f
I20260927 07:06:36.745694 1435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e: Bootstrap starting.
Sep 27 07:06:36 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:36.749598 1435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:36.751053 1435 log.cc:824] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:36.754626 1435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e: No bootstrap required, opened a new log
I20260927 07:06:36.757561 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:36.734964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.30.16.126:42729)
0927 07:06:36.735868 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:36.735881 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:36.735928 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:06:36.737646 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:36.737671 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:36.738127 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:36.738487 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:36.740170 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:36.741580 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:36.743372 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:36.744100 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:36.744289 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:36.748903 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:36.748914 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:36.748933 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:36.748984 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:36.752928 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:36.753679 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:36.753690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:36.753701 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:36.753904 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:36.754286 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:36.754290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:36.754293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:36.755033 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:36.755393 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:36.755688 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 07:06:36.756163 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":643,"thread_start_us":415,"threads_started":1}
I20260927 07:06:36.763578 1435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } }
I20260927 07:06:36.763917 1435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:36.763986 1435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46019cae98c546e5b8581a4bd55ae49e, State: Initialized, Role: FOLLOWER
I20260927 07:06:36.764456 1435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } }
I20260927 07:06:36.764643 1435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:36.764730 1435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:36.764882 1435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:36.767351 1435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } }
I20260927 07:06:36.767845 1435 leader_election.cc:302] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46019cae98c546e5b8581a4bd55ae49e; no voters:
I20260927 07:06:36.768543 1435 leader_election.cc:288] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:36.768713 1440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:36.769855 1440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [term 1 LEADER]: Becoming Leader. State: Replica: 46019cae98c546e5b8581a4bd55ae49e, State: Running, Role: LEADER
I20260927 07:06:36.770438 1440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } }
I20260927 07:06:36.771020 1435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:36.773579 1442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46019cae98c546e5b8581a4bd55ae49e. Latest consensus state: current_term: 1 leader_uuid: "46019cae98c546e5b8581a4bd55ae49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } } }
I20260927 07:06:36.773916 1442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:36.774626 1441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46019cae98c546e5b8581a4bd55ae49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46019cae98c546e5b8581a4bd55ae49e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42729 } } }
I20260927 07:06:36.775895 1441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:36.776034 1447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:36.781275 1447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:36.791651 1447 catalog_manager.cc:1347] Generated new cluster ID: ca573339d9db42318ac52ad0aabcfb81
I20260927 07:06:36.791791 1447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:36.808487 1447 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:36.810022 1447 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:36.819231 1447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46019cae98c546e5b8581a4bd55ae49e: Generated new TSK 0
I20260927 07:06:36.820307 1447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:36.882808 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:37.240614 1461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:37.240934 1461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:37.241030 1461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:37.248420 1461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:37.248538 1461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:37.248652 1461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:37.259119 1461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:37.260397 1461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:37.261651 1461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:37.275460 1469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:37.275506 1467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:37.275565 1466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:37.276592 1461 server_base.cc:1034] running on GCE node
I20260927 07:06:37.277364 1461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:37.278347 1461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:37.279558 1461 hybrid_clock.cc:648] HybridClock initialized: now 1790492797279490 us; error 54 us; skew 500 ppm
I20260927 07:06:37.279954 1461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:37 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:37.293695 1461 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:37.295498 1461 webserver.cc:460] Webserver started at http://127.30.16.65:42257/ using document root <none> and password file <none>
I20260927 07:06:37.296275 1461 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:37.296437 1461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:37.296753 1461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:37.299911 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "68954f323bbc450f885a574f56e023a0"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "5cda6cb7a3cce103d58f33fe913f2de3"
server_key_iv: "8886518928ad8deec20e84fb16463b11"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.300740 1461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "68954f323bbc450f885a574f56e023a0"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "5cda6cb7a3cce103d58f33fe913f2de3"
server_key_iv: "8886518928ad8deec20e84fb16463b11"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.306532 1461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:37.310333 1476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.311872 1461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:37.312214 1461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "68954f323bbc450f885a574f56e023a0"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "5cda6cb7a3cce103d58f33fe913f2de3"
server_key_iv: "8886518928ad8deec20e84fb16463b11"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.312433 1461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:37.332135 1461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:37.336639 1461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:37.336961 1461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:37.338035 1461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:37.339567 1461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:37.339671 1461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.339829 1461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:37.339905 1461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.374126 1461 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:44149
I20260927 07:06:37.374172 1588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:44149 every 8 connection(s)
I20260927 07:06:37.375480 1461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:37.385128 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1461
I20260927 07:06:37.385331 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:37.385975 30785 external_mini_cluster.cc:1428] Setting key 76f0469d89e6cb29ffa519d4bb1507c9
Sep 27 07:06:37 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:37.398176 1591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:37.378049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:43911 (local address 127.30.16.126:42729)
0927 07:06:37.378498 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:37.378505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:37.379698 (+ 1193us) server_negotiation.cc:407] Connection header received
0927 07:06:37.381689 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:37.381693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:37.381768 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:37.381867 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:37.383665 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.384388 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.386215 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.386749 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.386882 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:37.391524 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:37.391530 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:37.391534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:37.391570 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:37.394414 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:37.395322 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:37.395331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:37.395338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:37.395495 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:37.395928 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:37.395936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:37.395942 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:37.396339 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:37.396489 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:37.397387 (+ 898us) server_negotiation.cc:299] Negotiation successful
0927 07:06:37.397670 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260927 07:06:37.399243 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:37.378737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42729 (local address 127.30.16.65:43911)
0927 07:06:37.379541 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:37.379587 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:37.381439 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:37.382150 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:37.382159 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:37.382879 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:37.383505 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:37.383524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.384543 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.384547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:37.386038 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:37.386055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.386991 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.386999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:37.387246 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:37.388045 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:37.388082 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:37.391210 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:37.394652 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:37.394664 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:37.394688 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:37.395167 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:37.395611 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:37.395615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:37.395617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:37.395806 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:37.396464 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:37.396470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:37.396822 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 07:06:37.397265 (+ 443us) client_negotiation.cc:240] Negotiation successful
0927 07:06:37.397979 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":276,"threads_started":1}
I20260927 07:06:37.401001 1589 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42729
I20260927 07:06:37.401458 1589 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:37.402233 1589 heartbeater.cc:511] Master 127.30.16.126:42729 requested a full tablet report, sending...
I20260927 07:06:37.404479 1400 ts_manager.cc:194] Registered new tserver with Master: 68954f323bbc450f885a574f56e023a0 (127.30.16.65:44149)
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:37.406286 1400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:43911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:37.451136 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:37.779623 1595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:37.779953 1595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:37.780121 1595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:37.787489 1595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:37.787633 1595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:37.787788 1595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:37.797526 1595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:37.798790 1595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:37.800087 1595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:37.813586 1601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:37.813630 1603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:37.813611 1600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:37.814471 1595 server_base.cc:1034] running on GCE node
I20260927 07:06:37.815131 1595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:37.816186 1595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:37.817410 1595 hybrid_clock.cc:648] HybridClock initialized: now 1790492797817358 us; error 46 us; skew 500 ppm
I20260927 07:06:37.817727 1595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:37 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:37.820806 1595 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:37.822273 1595 webserver.cc:460] Webserver started at http://127.30.16.66:42719/ using document root <none> and password file <none>
I20260927 07:06:37.823061 1595 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:37.823206 1595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:37.823518 1595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:37.826493 1595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "8a69f8f99b2942298fcb5e66855ab21e"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "9bc6d75d4a2f7f6cacc120da1bacded7"
server_key_iv: "3b438df3be582a5c59a00cb8c8869b39"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.827354 1595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "8a69f8f99b2942298fcb5e66855ab21e"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "9bc6d75d4a2f7f6cacc120da1bacded7"
server_key_iv: "3b438df3be582a5c59a00cb8c8869b39"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.832588 1595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 07:06:37.836057 1610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.837397 1595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:06:37.837579 1595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "8a69f8f99b2942298fcb5e66855ab21e"
format_stamp: "Formatted at 2026-09-27 07:06:37 on dist-test-slave-8vqf"
server_key: "9bc6d75d4a2f7f6cacc120da1bacded7"
server_key_iv: "3b438df3be582a5c59a00cb8c8869b39"
server_key_version: "encryptionkey@0"
I20260927 07:06:37.837777 1595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:37.855182 1595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:37.859680 1595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:37.859992 1595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:37.860998 1595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:37.862416 1595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:37.862560 1595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.862720 1595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:37.862804 1595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:37.896523 1595 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:41697
I20260927 07:06:37.896566 1722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:41697 every 8 connection(s)
I20260927 07:06:37.897831 1595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:37.900367 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1595
I20260927 07:06:37.900556 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:37.901144 30785 external_mini_cluster.cc:1428] Setting key b1ecfd776005554686eb0af03186f4fd
Sep 27 07:06:37 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:37.918727 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:37.900556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:32863 (local address 127.30.16.126:42729)
0927 07:06:37.901154 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:37.901158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:37.902379 (+ 1221us) server_negotiation.cc:407] Connection header received
0927 07:06:37.904412 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:37.904418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:37.904501 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:37.904628 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:37.906228 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.907043 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.908622 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.909065 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.909205 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:37.913009 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:37.913015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:37.913018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:37.913050 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:37.915615 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:37.916375 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:37.916386 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:37.916389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:37.916524 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:37.916965 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:37.916972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:37.916977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:37.917233 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:37.917441 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:37.918090 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 07:06:37.918302 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":366,"threads_started":1}
I20260927 07:06:37.919790 1726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:37.901275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42729 (local address 127.30.16.66:32863)
0927 07:06:37.902217 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:37.902276 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:37.904137 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:37.904894 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:37.904903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:37.905455 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:37.906040 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:37.906057 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.907210 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.907220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:37.908465 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:37.908476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:37.909295 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:37.909303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:37.909546 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:37.910253 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:37.910276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:37.912770 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:37.915790 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:37.915800 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:37.915818 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:37.916264 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:37.916635 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:37.916640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:37.916643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:37.916854 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:37.917360 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:37.917368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:37.917740 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 07:06:37.918021 (+ 281us) client_negotiation.cc:240] Negotiation successful
0927 07:06:37.918578 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":296,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:37.921332 1723 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42729
I20260927 07:06:37.921670 1723 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:37.922427 1723 heartbeater.cc:511] Master 127.30.16.126:42729 requested a full tablet report, sending...
I20260927 07:06:37.924014 1400 ts_manager.cc:194] Registered new tserver with Master: 8a69f8f99b2942298fcb5e66855ab21e (127.30.16.66:41697)
I20260927 07:06:37.924850 1400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:32863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:37.946506 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:38.279804 1729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:38.280160 1729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:38.280260 1729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:38.287798 1729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:38.287928 1729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:38.288043 1729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:38.298243 1729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42729
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:38.299798 1729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:38.301115 1729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:38.315200 1734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:38.315276 1737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:38.315236 1735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:38.315690 1729 server_base.cc:1034] running on GCE node
I20260927 07:06:38.316565 1729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:38.317600 1729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:38.318796 1729 hybrid_clock.cc:648] HybridClock initialized: now 1790492798318738 us; error 52 us; skew 500 ppm
I20260927 07:06:38.319133 1729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:38.322290 1729 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:38.323995 1729 webserver.cc:460] Webserver started at http://127.30.16.67:40677/ using document root <none> and password file <none>
I20260927 07:06:38.324723 1729 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:38.324869 1729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:38.325178 1729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:38.328248 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "50e961e6366443aa85516677ad699557"
format_stamp: "Formatted at 2026-09-27 07:06:38 on dist-test-slave-8vqf"
server_key: "34bf676c1a4c4862cf29cd94e97a6edb"
server_key_iv: "e883f8f3de002ab771ad98839162a8cc"
server_key_version: "encryptionkey@0"
I20260927 07:06:38.329183 1729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "50e961e6366443aa85516677ad699557"
format_stamp: "Formatted at 2026-09-27 07:06:38 on dist-test-slave-8vqf"
server_key: "34bf676c1a4c4862cf29cd94e97a6edb"
server_key_iv: "e883f8f3de002ab771ad98839162a8cc"
server_key_version: "encryptionkey@0"
I20260927 07:06:38.334510 1729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 07:06:38.337947 1744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:38.339351 1729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:38.339545 1729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "50e961e6366443aa85516677ad699557"
format_stamp: "Formatted at 2026-09-27 07:06:38 on dist-test-slave-8vqf"
server_key: "34bf676c1a4c4862cf29cd94e97a6edb"
server_key_iv: "e883f8f3de002ab771ad98839162a8cc"
server_key_version: "encryptionkey@0"
I20260927 07:06:38.339743 1729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:38.352713 1729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:38.357126 1729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:38.357451 1729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:38.358402 1729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:38.359874 1729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:38.359992 1729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:38.360081 1729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:38.360128 1729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:38.396425 1729 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:44433
I20260927 07:06:38.396461 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:44433 every 8 connection(s)
I20260927 07:06:38.397733 1729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:38.405584 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1729
I20260927 07:06:38.405786 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:38.406266 30785 external_mini_cluster.cc:1428] Setting key 1e954d4630666248e503e7bec35044f1
I20260927 07:06:38.410086 1589 heartbeater.cc:503] Master 127.30.16.126:42729 was elected leader, sending a full tablet report...
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:38.419164 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.400609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:41045 (local address 127.30.16.126:42729)
0927 07:06:38.400781 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:38.400785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:38.401865 (+ 1080us) server_negotiation.cc:407] Connection header received
0927 07:06:38.403841 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:38.403847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:38.403955 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:38.404065 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:38.405883 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.406830 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.408903 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.409526 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.409627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.413462 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:38.413473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:38.413476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:38.413507 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:38.416088 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.416784 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.416791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.416795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.416926 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.417415 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.417420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.417423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.417707 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:38.417872 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:38.418594 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 07:06:38.418820 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:06:38.420209 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.400997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42729 (local address 127.30.16.67:41045)
0927 07:06:38.401703 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:38.401745 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:38.403519 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:38.404307 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:38.404316 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:38.404840 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:38.405644 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:38.405672 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.406994 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.406998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:38.408772 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:38.408782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.409737 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.409743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:38.409934 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.410660 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:38.410681 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:38.413225 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:38.416227 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:38.416236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:38.416254 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:38.416665 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:38.417038 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:38.417043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:38.417047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:38.417297 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:38.417835 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:38.417844 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:38.418203 (+ 359us) client_negotiation.cc:769] Sending connection context
0927 07:06:38.418481 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 07:06:38.419073 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":268,"threads_started":1}
I20260927 07:06:38.421711 1857 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42729
I20260927 07:06:38.422041 1857 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:38.422823 1857 heartbeater.cc:511] Master 127.30.16.126:42729 requested a full tablet report, sending...
I20260927 07:06:38.424196 1400 ts_manager.cc:194] Registered new tserver with Master: 50e961e6366443aa85516677ad699557 (127.30.16.67:44433)
I20260927 07:06:38.424930 1400 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:41045
I20260927 07:06:38.434435 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:38.481848 1725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.467782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44008 (local address 127.30.16.126:42729)
0927 07:06:38.467957 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:38.467983 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:38.468467 (+ 484us) server_negotiation.cc:407] Connection header received
0927 07:06:38.468694 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:38.468699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:38.468763 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:38.468865 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:38.470213 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.471083 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.472533 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.473009 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.473110 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.476809 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:38.476814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:38.476817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:38.476850 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:38.479523 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.480142 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.480151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.480153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.480264 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.480627 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.480631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.480632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.480924 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:38.481122 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:38.481342 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 07:06:38.481506 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 07:06:38.482388 1400 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44008
I20260927 07:06:38.483955 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1461
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:38.499863 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.485234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59022 (local address 127.30.16.65:44149)
0927 07:06:38.485789 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:38.485796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:38.485819 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:06:38.485871 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:38.485877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:38.486113 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:38.486233 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:38.487358 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.488329 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.489502 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.490058 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.490152 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.494185 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:38.494196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:38.494211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:38.494261 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:38.497290 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.497924 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.497931 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.497933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.498046 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.498362 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.498366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.498367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.498809 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:38.499035 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:38.499257 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:06:38.499458 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
W20260927 07:06:38.501168 1584 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59022
W20260927 07:06:38.502424 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1461: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:38.502681 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1461
I20260927 07:06:38.518251 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1595
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:38.535566 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.519301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.30.16.66:41697)
0927 07:06:38.519846 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:38.519853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:38.519878 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:06:38.519943 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:38.519972 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:38.520180 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:38.520308 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:38.521485 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.522462 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.523772 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.524316 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.524413 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.528678 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:38.528708 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:38.528744 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:38.528803 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:38.532191 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.532818 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.532825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.532827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.533014 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.533367 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.533371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.533373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.533874 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:38.534246 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:38.534786 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 07:06:38.535168 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":296,"threads_started":1}
W20260927 07:06:38.537096 1718 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35666
W20260927 07:06:38.538599 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1595: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:38.538784 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1595
I20260927 07:06:38.555557 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1729
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492798, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:38.572423 1870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:38.556536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.30.16.67:44433)
0927 07:06:38.557165 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:38.557172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:38.557201 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:06:38.557265 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:38.557272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:38.557520 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:38.557693 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:38.559110 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.560135 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.561366 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:38.561941 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:38.562080 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:38.566869 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:38.566880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:38.566895 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:38.566935 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:38.569573 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.570287 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.570294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.570296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.570412 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:38.570796 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:38.570802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:38.570805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:38.571156 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:38.571431 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:38.571731 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 07:06:38.572018 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":331,"threads_started":1}
W20260927 07:06:38.573786 1852 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37378
W20260927 07:06:38.574981 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:06:38.575129 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1729
I20260927 07:06:38.590976 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1368
I20260927 07:06:38.592253 1430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:38.695544 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1368
2026-09-27T07:06:38Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3708 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1874](info): set up 2 sockets
Sep 27 07:06:38 dist-test-slave-8vqf krb5kdc[1874](info): commencing operation
krb5kdc: starting...
W20260927 07:06:39.799014 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:39 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:40.003132 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43211
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43211
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:40.332804 1887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:40.333165 1887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:40.333261 1887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:40.340742 1887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:40.340860 1887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:40.340946 1887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:40.341045 1887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:40.341104 1887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:40.351289 1887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:40.352701 1887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:40.354105 1887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:40.366341 1895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:40.366297 1892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:40.366376 1893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:40.367125 1887 server_base.cc:1034] running on GCE node
I20260927 07:06:40.367941 1887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:40.369501 1887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:40.370690 1887 hybrid_clock.cc:648] HybridClock initialized: now 1790492800370642 us; error 55 us; skew 500 ppm
I20260927 07:06:40.371101 1887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:40 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492800, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:40.374327 1887 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:40.376142 1887 webserver.cc:460] Webserver started at http://127.30.16.126:37223/ using document root <none> and password file <none>
I20260927 07:06:40.376875 1887 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:40.377017 1887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:40.377332 1887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:40.380496 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "4a0f49c2ef4a4216b55ee80daab57e78"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "631775061d08ec14914ccaac5c4125d5"
server_key_iv: "d9df0dbc7b69f04f2494cb2f1877dedb"
server_key_version: "encryptionkey@0"
I20260927 07:06:40.381294 1887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "4a0f49c2ef4a4216b55ee80daab57e78"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "631775061d08ec14914ccaac5c4125d5"
server_key_iv: "d9df0dbc7b69f04f2494cb2f1877dedb"
server_key_version: "encryptionkey@0"
I20260927 07:06:40.386639 1887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 07:06:40.390215 1902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:40.391672 1887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:06:40.391865 1887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "4a0f49c2ef4a4216b55ee80daab57e78"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "631775061d08ec14914ccaac5c4125d5"
server_key_iv: "d9df0dbc7b69f04f2494cb2f1877dedb"
server_key_version: "encryptionkey@0"
I20260927 07:06:40.392103 1887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:40.419251 1887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:40.423645 1887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:40.423941 1887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:40.442469 1887 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43211
I20260927 07:06:40.442510 1953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43211 every 8 connection(s)
I20260927 07:06:40.443964 1887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:40.447883 1954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:40.452824 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1887
I20260927 07:06:40.453074 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:40.453866 30785 external_mini_cluster.cc:1428] Setting key 493d5f2c3722c63ebb66e086766b0fff
I20260927 07:06:40.459441 1954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78: Bootstrap starting.
I20260927 07:06:40.463233 1954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:40.464533 1954 log.cc:824] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:40.468020 1954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78: No bootstrap required, opened a new log
Sep 27 07:06:40 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:40.476522 1954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } }
I20260927 07:06:40.476898 1954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:40.477020 1954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a0f49c2ef4a4216b55ee80daab57e78, State: Initialized, Role: FOLLOWER
I20260927 07:06:40.477695 1954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } }
I20260927 07:06:40.477947 1954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:40.478221 1954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:40.478477 1954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:40.479712 1957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:40.456183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.30.16.126:43211)
0927 07:06:40.457114 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:40.457130 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:40.457176 (+ 46us) server_negotiation.cc:407] Connection header received
0927 07:06:40.459126 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:40.459149 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:40.459621 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:40.460008 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:40.461869 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:40.463480 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:40.465355 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:40.466392 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:40.466539 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:40.471413 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:40.471421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:40.471437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:40.471474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:40.475048 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:40.475814 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:40.475825 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:40.475835 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:40.476018 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:40.476442 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:40.476452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:40.476454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:40.477148 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:40.477473 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:40.477969 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 07:06:40.478488 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":437,"threads_started":1}
I20260927 07:06:40.481091 1954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } }
I20260927 07:06:40.481588 1954 leader_election.cc:302] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a0f49c2ef4a4216b55ee80daab57e78; no voters:
I20260927 07:06:40.482182 1954 leader_election.cc:288] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:40.482343 1959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:40.483798 1959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [term 1 LEADER]: Becoming Leader. State: Replica: 4a0f49c2ef4a4216b55ee80daab57e78, State: Running, Role: LEADER
I20260927 07:06:40.484398 1954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:40.484335 1959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } }
I20260927 07:06:40.487115 1960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } } }
I20260927 07:06:40.487574 1960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:40.487474 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a0f49c2ef4a4216b55ee80daab57e78. Latest consensus state: current_term: 1 leader_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a0f49c2ef4a4216b55ee80daab57e78" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43211 } } }
I20260927 07:06:40.487804 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:40.488757 1968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:40.496224 1968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:40.506687 1968 catalog_manager.cc:1347] Generated new cluster ID: a957bb311994499fbf1e3ec546bf7f4e
I20260927 07:06:40.506814 1968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:40.517802 1968 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:40.519867 1968 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:40.540428 1968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a0f49c2ef4a4216b55ee80daab57e78: Generated new TSK 0
I20260927 07:06:40.541765 1968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:40.603647 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:40.937721 1980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:40.938037 1980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:40.938201 1980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:40.945608 1980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:40.945727 1980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:40.945845 1980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:40.956171 1980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:40.957576 1980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:40.958963 1980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:40.975766 1985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:40.975696 1988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:40.975775 1986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:40.976527 1980 server_base.cc:1034] running on GCE node
I20260927 07:06:40.977243 1980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:40.978199 1980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:40.979429 1980 hybrid_clock.cc:648] HybridClock initialized: now 1790492800979382 us; error 37 us; skew 500 ppm
I20260927 07:06:40.979732 1980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:40 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492800, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:40.983019 1980 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:40.984601 1980 webserver.cc:460] Webserver started at http://127.30.16.65:37771/ using document root <none> and password file <none>
I20260927 07:06:40.985311 1980 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:40.985457 1980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:40.985765 1980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:40.988993 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "2aae3b25eb384f399147b4e136a332d1"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "a6fa7dca2f4ced42dbace2d8a6776ddd"
server_key_iv: "90039c5cf88633a064a4c51a6423f158"
server_key_version: "encryptionkey@0"
I20260927 07:06:40.989816 1980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "2aae3b25eb384f399147b4e136a332d1"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "a6fa7dca2f4ced42dbace2d8a6776ddd"
server_key_iv: "90039c5cf88633a064a4c51a6423f158"
server_key_version: "encryptionkey@0"
I20260927 07:06:40.995514 1980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:40.999419 1995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.000955 1980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:41.001148 1980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "2aae3b25eb384f399147b4e136a332d1"
format_stamp: "Formatted at 2026-09-27 07:06:40 on dist-test-slave-8vqf"
server_key: "a6fa7dca2f4ced42dbace2d8a6776ddd"
server_key_iv: "90039c5cf88633a064a4c51a6423f158"
server_key_version: "encryptionkey@0"
I20260927 07:06:41.001359 1980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:41.037834 1980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:41.042294 1980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:41.042665 1980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:41.043753 1980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:41.045065 1980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:41.045145 1980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.045298 1980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:41.045356 1980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.079191 1980 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45961
I20260927 07:06:41.079231 2107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45961 every 8 connection(s)
I20260927 07:06:41.080577 1980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:41.083784 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 1980
I20260927 07:06:41.084046 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:41.084918 30785 external_mini_cluster.cc:1428] Setting key 8cd057e00566c768f186c8f28c5d47f7
Sep 27 07:06:41 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492800, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:41.103098 2110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:41.083321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:46039 (local address 127.30.16.126:43211)
0927 07:06:41.083848 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:41.083856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:41.085098 (+ 1242us) server_negotiation.cc:407] Connection header received
0927 07:06:41.087471 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:41.087476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:41.087538 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:41.087633 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:41.089817 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.090619 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.092389 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.092872 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.093003 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:41.096988 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:41.096993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:41.096997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:41.097030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:41.099842 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:41.100612 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:41.100620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:41.100624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:41.100752 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:41.101191 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:41.101199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:41.101202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:41.101478 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:41.101633 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:41.102383 (+ 750us) server_negotiation.cc:299] Negotiation successful
0927 07:06:41.102674 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":244,"threads_started":1}
I20260927 07:06:41.104327 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:41.083996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43211 (local address 127.30.16.65:46039)
0927 07:06:41.084899 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:41.084960 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:41.087161 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:41.087898 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:41.087916 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:41.088759 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:41.089626 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:41.089660 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.090802 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.090806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:41.092238 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:41.092265 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.093094 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.093103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:41.093342 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:41.094116 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:41.094141 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:41.096738 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:41.099989 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:41.099999 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:41.100018 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:41.100485 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:41.100871 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:41.100878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:41.100882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:41.101077 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:41.101606 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:41.101614 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:41.101997 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 07:06:41.102303 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 07:06:41.102929 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":355,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:06:41.106101 2108 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43211
I20260927 07:06:41.106436 2108 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:41.107224 2108 heartbeater.cc:511] Master 127.30.16.126:43211 requested a full tablet report, sending...
I20260927 07:06:41.109440 1919 ts_manager.cc:194] Registered new tserver with Master: 2aae3b25eb384f399147b4e136a332d1 (127.30.16.65:45961)
I20260927 07:06:41.111199 1919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:46039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:41.130290 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:41.472290 2114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:41.472604 2114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:41.472757 2114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:41.480585 2114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:41.480722 2114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:41.480917 2114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:41.491343 2114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:41.492697 2114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:41.494066 2114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:41.507961 2122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:41.508065 2120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:41.508236 2119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:41.508742 2114 server_base.cc:1034] running on GCE node
I20260927 07:06:41.509384 2114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:41.510426 2114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:41.511642 2114 hybrid_clock.cc:648] HybridClock initialized: now 1790492801511598 us; error 41 us; skew 500 ppm
I20260927 07:06:41.511978 2114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:41 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492801, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:41.515246 2114 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:41.516813 2114 webserver.cc:460] Webserver started at http://127.30.16.66:36183/ using document root <none> and password file <none>
I20260927 07:06:41.517539 2114 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:41.517669 2114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:41.518023 2114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:41.521099 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "6335814f39f84540b6e6dd24782b330a"
format_stamp: "Formatted at 2026-09-27 07:06:41 on dist-test-slave-8vqf"
server_key: "7ff0fbd0a76ac320a170fefd373afc4d"
server_key_iv: "42a0008e52ad5e2ce51fad490d138e37"
server_key_version: "encryptionkey@0"
I20260927 07:06:41.521903 2114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "6335814f39f84540b6e6dd24782b330a"
format_stamp: "Formatted at 2026-09-27 07:06:41 on dist-test-slave-8vqf"
server_key: "7ff0fbd0a76ac320a170fefd373afc4d"
server_key_iv: "42a0008e52ad5e2ce51fad490d138e37"
server_key_version: "encryptionkey@0"
I20260927 07:06:41.527411 2114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 07:06:41.530958 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.532529 2114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:41.532757 2114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "6335814f39f84540b6e6dd24782b330a"
format_stamp: "Formatted at 2026-09-27 07:06:41 on dist-test-slave-8vqf"
server_key: "7ff0fbd0a76ac320a170fefd373afc4d"
server_key_iv: "42a0008e52ad5e2ce51fad490d138e37"
server_key_version: "encryptionkey@0"
I20260927 07:06:41.533001 2114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:41.563520 2114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:41.568598 2114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:41.568893 2114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:41.569921 2114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:41.571379 2114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:41.571475 2114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.571611 2114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:41.571679 2114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:41.604861 2114 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:41643
I20260927 07:06:41.604991 2241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:41643 every 8 connection(s)
I20260927 07:06:41.606292 2114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:41.608480 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2114
I20260927 07:06:41.608687 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:41.609274 30785 external_mini_cluster.cc:1428] Setting key 55dad1fa8d40e90a8b5ad4d71d10d667
Sep 27 07:06:41 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492801, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:41.627333 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:41.609243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:60515 (local address 127.30.16.126:43211)
0927 07:06:41.609754 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:41.609759 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:41.611066 (+ 1307us) server_negotiation.cc:407] Connection header received
0927 07:06:41.613100 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:41.613105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:41.613190 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:41.613299 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:41.615035 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.615748 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.617330 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.617768 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.617900 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:41.621368 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:41.621373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:41.621376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:41.621408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:41.624022 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:41.624836 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:41.624840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:41.624842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:41.624982 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:41.625410 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:41.625415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:41.625418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:41.625754 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:41.625902 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:41.626693 (+ 791us) server_negotiation.cc:299] Negotiation successful
0927 07:06:41.626962 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":304,"threads_started":1}
I20260927 07:06:41.628492 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:41.609928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43211 (local address 127.30.16.66:60515)
0927 07:06:41.610889 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:41.610943 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:41.612811 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:41.613574 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:41.613584 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:41.614190 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:41.614862 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:41.614881 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.615897 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.615902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:41.617182 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:41.617194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:41.617991 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:41.618029 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:41.618257 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:41.618965 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:41.618989 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:41.621139 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:41.624167 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:41.624176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:41.624194 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:41.624675 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:41.625094 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:41.625099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:41.625103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:41.625292 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:41.625902 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:41.625911 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:41.626283 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 07:06:41.626593 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 07:06:41.627170 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":414,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:41.630260 2242 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43211
I20260927 07:06:41.630656 2242 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:41.631397 2242 heartbeater.cc:511] Master 127.30.16.126:43211 requested a full tablet report, sending...
I20260927 07:06:41.633049 1919 ts_manager.cc:194] Registered new tserver with Master: 6335814f39f84540b6e6dd24782b330a (127.30.16.66:41643)
I20260927 07:06:41.634374 1919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:60515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:41.656236 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:41.991822 2248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:41.992179 2248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:41.992344 2248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:42.000236 2248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:42.000372 2248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:42.000560 2248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:42.011416 2248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43211
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:42.013037 2248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:42.014434 2248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:42.028558 2253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:42.028712 2256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:42.028496 2254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:42.029686 2248 server_base.cc:1034] running on GCE node
I20260927 07:06:42.030443 2248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:42.031474 2248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:42.032697 2248 hybrid_clock.cc:648] HybridClock initialized: now 1790492802032624 us; error 62 us; skew 500 ppm
I20260927 07:06:42.033062 2248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:42.036453 2248 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:42.038034 2248 webserver.cc:460] Webserver started at http://127.30.16.67:39389/ using document root <none> and password file <none>
I20260927 07:06:42.038846 2248 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:42.038985 2248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:42.039304 2248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:42.042201 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "2f1325596f854bf9a2d40a4f60bc98a7"
format_stamp: "Formatted at 2026-09-27 07:06:42 on dist-test-slave-8vqf"
server_key: "847c7889e96190ec318c6d4cb3fc654c"
server_key_iv: "8de767fc013a2794d405e66511f124e3"
server_key_version: "encryptionkey@0"
I20260927 07:06:42.043128 2248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "2f1325596f854bf9a2d40a4f60bc98a7"
format_stamp: "Formatted at 2026-09-27 07:06:42 on dist-test-slave-8vqf"
server_key: "847c7889e96190ec318c6d4cb3fc654c"
server_key_iv: "8de767fc013a2794d405e66511f124e3"
server_key_version: "encryptionkey@0"
I20260927 07:06:42.048499 2248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:42.052096 2263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:42.053581 2248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:06:42.053773 2248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "2f1325596f854bf9a2d40a4f60bc98a7"
format_stamp: "Formatted at 2026-09-27 07:06:42 on dist-test-slave-8vqf"
server_key: "847c7889e96190ec318c6d4cb3fc654c"
server_key_iv: "8de767fc013a2794d405e66511f124e3"
server_key_version: "encryptionkey@0"
I20260927 07:06:42.053988 2248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:42.078948 2248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:42.084358 2248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:42.084681 2248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:42.085809 2248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:42.087131 2248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:42.087201 2248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:42.087285 2248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:42.087328 2248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:42.114516 2108 heartbeater.cc:503] Master 127.30.16.126:43211 was elected leader, sending a full tablet report...
I20260927 07:06:42.121026 2248 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:41509
I20260927 07:06:42.121079 2375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:41509 every 8 connection(s)
I20260927 07:06:42.122352 2248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:42.126181 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2248
I20260927 07:06:42.126384 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:42.126945 30785 external_mini_cluster.cc:1428] Setting key ae5652a3c34bbac61ba6476699d64f66
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:42.145143 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.124963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:49549 (local address 127.30.16.126:43211)
0927 07:06:42.125188 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.125195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.126675 (+ 1480us) server_negotiation.cc:407] Connection header received
0927 07:06:42.128868 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.128874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.128999 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.129103 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.131147 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.132394 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.133990 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.134689 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.134839 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.139102 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.139108 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.139112 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.139150 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.142468 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.143198 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.143202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.143205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.143321 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.143654 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.143657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.143659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.143917 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.144032 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.144695 (+ 663us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.144909 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:06:42.146179 2378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.125628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43211 (local address 127.30.16.67:49549)
0927 07:06:42.126471 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:42.126525 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:42.128577 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:42.129393 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:42.129405 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:42.130104 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:42.130956 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:42.130983 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.132549 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.132555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:42.133839 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:42.133850 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.134932 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.134943 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:42.135297 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.136112 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:42.136134 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:42.138859 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:42.142652 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:42.142659 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:42.142673 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:42.143093 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:42.143394 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:42.143397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:42.143399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:42.143566 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:42.144032 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:42.144038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:42.144359 (+ 321us) client_negotiation.cc:769] Sending connection context
0927 07:06:42.144600 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 07:06:42.144988 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":357,"threads_started":1}
I20260927 07:06:42.147496 2376 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43211
I20260927 07:06:42.147805 2376 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:42.148517 2376 heartbeater.cc:511] Master 127.30.16.126:43211 requested a full tablet report, sending...
I20260927 07:06:42.149983 1919 ts_manager.cc:194] Registered new tserver with Master: 2f1325596f854bf9a2d40a4f60bc98a7 (127.30.16.67:41509)
I20260927 07:06:42.150915 1919 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:49549
I20260927 07:06:42.154626 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:42.203387 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.188334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.30.16.65:45961)
0927 07:06:42.189002 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.189009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.189035 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:42.189107 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.189114 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.189325 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.189522 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.191072 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.192094 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.193355 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.193970 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.194120 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.198013 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.198025 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.198040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.198079 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.200925 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.201615 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.201622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.201624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.201747 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.202064 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.202067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.202069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.202399 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.202747 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.202878 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.203087 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":308,"threads_started":1}
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:42.221367 2244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.207914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.30.16.126:43211)
0927 07:06:42.208140 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.208147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.208289 (+ 142us) server_negotiation.cc:407] Connection header received
0927 07:06:42.208488 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.208493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.208575 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.208675 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.210253 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.211067 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.212296 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.212717 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.212855 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.216709 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.216714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.216717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.216751 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.219176 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.219730 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.219734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.219736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.219851 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.220163 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.220166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.220167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.220463 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.220638 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.220899 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.221100 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 07:06:42.221905 1919 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37368
I20260927 07:06:42.222988 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1980
I20260927 07:06:42.235912 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.223870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.30.16.65:45961)
0927 07:06:42.224034 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.224038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.224244 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:06:42.224381 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.224385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.224456 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.224532 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.225659 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.226426 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.228115 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.228789 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.228932 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.231418 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.231422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.231426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.231460 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.233770 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.234318 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.234322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.234324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.234433 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.234775 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.234778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.234781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.235018 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.235184 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.235439 (+ 255us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.235594 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:06:42.236531 2103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:42.320153 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1980
I20260927 07:06:42.336997 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2114
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:42.362057 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.338069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44430 (local address 127.30.16.66:41643)
0927 07:06:42.338696 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.338709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.338735 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:42.338805 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.338811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.339044 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.339177 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.340554 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.341561 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.343222 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.343838 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.343972 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.355787 (+ 11815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.355804 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.355821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.355864 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.359142 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.359983 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.359991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.359993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.360119 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.360530 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.360534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.360536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.360882 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.361112 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.361446 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.361692 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
I20260927 07:06:42.363401 2237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:42.448063 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2114
I20260927 07:06:42.467859 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2248
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[1874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:42.484661 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:42.468771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.30.16.67:41509)
0927 07:06:42.469355 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:42.469362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:42.469391 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:06:42.469459 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:42.469465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:42.469677 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:42.469806 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:42.471180 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.472203 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.473405 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:42.473987 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:42.474086 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:42.478693 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:42.478705 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:42.478721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:42.478763 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:42.481605 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.482341 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.482348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.482350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.482473 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:42.482927 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:42.482935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:42.482938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:42.483359 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:42.483683 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:42.483994 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 07:06:42.484302 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":272,"threads_started":1}
I20260927 07:06:42.486102 2371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:42.574244 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2248
I20260927 07:06:42.590958 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 1887
I20260927 07:06:42.592254 1949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:42.694370 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 1887
2026-09-27T07:06:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3998 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[2395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[2395](info): set up 2 sockets
Sep 27 07:06:42 dist-test-slave-8vqf krb5kdc[2395](info): commencing operation
krb5kdc: starting...
W20260927 07:06:43.831568 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:43 dist-test-slave-8vqf krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:44.039391 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39117
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:44.377490 2408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:44.377872 2408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:44.378050 2408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:44.385545 2408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:44.385723 2408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:44.385840 2408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:44.385944 2408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:44.386035 2408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:44.397368 2408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:44.398767 2408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:44.400122 2408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:44.411536 2414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:44.411598 2416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:44.411616 2413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:44.412539 2408 server_base.cc:1034] running on GCE node
I20260927 07:06:44.413401 2408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:44.415058 2408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:44.416321 2408 hybrid_clock.cc:648] HybridClock initialized: now 1790492804416292 us; error 44 us; skew 500 ppm
I20260927 07:06:44.416721 2408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:44 dist-test-slave-8vqf krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492804, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:44.419916 2408 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:44.421039 2408 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:44.421126 2408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:44.421380 2408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:44.424427 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "3f23f3ef74f845b1bb3460cb756ab709"
format_stamp: "Formatted at 2026-09-27 07:06:44 on dist-test-slave-8vqf"
server_key: "b8005fcb4f8cdef1f5e39b1fe37afbc5"
server_key_iv: "3891731f00cef9588b7e71288d425128"
server_key_version: "encryptionkey@0"
I20260927 07:06:44.425181 2408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "3f23f3ef74f845b1bb3460cb756ab709"
format_stamp: "Formatted at 2026-09-27 07:06:44 on dist-test-slave-8vqf"
server_key: "b8005fcb4f8cdef1f5e39b1fe37afbc5"
server_key_iv: "3891731f00cef9588b7e71288d425128"
server_key_version: "encryptionkey@0"
I20260927 07:06:44.430693 2408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 07:06:44.434717 2422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:44.436219 2408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:44.436383 2408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "3f23f3ef74f845b1bb3460cb756ab709"
format_stamp: "Formatted at 2026-09-27 07:06:44 on dist-test-slave-8vqf"
server_key: "b8005fcb4f8cdef1f5e39b1fe37afbc5"
server_key_iv: "3891731f00cef9588b7e71288d425128"
server_key_version: "encryptionkey@0"
I20260927 07:06:44.436610 2408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:44.461243 2408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:44.465662 2408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:44.465917 2408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:44.484616 2408 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:39117
I20260927 07:06:44.484635 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:39117 every 8 connection(s)
I20260927 07:06:44.485538 2408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:44.488885 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2408
I20260927 07:06:44.489104 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:44.489478 2474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:44.489890 30785 external_mini_cluster.cc:1428] Setting key 922a75e165a6f4dbdfc9b135c950d1ef
I20260927 07:06:44.504308 2474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: Bootstrap starting.
Sep 27 07:06:44 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:44.507830 2474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:44.508906 2474 log.cc:824] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:44.512303 2474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: No bootstrap required, opened a new log
I20260927 07:06:44.513294 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:44.491861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.30.16.126:39117)
0927 07:06:44.492848 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:44.492866 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:44.492909 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:06:44.494414 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:44.494436 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:44.494888 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:44.495266 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:44.497164 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:44.498470 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:44.500294 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:44.501041 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:44.501179 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:44.505848 (+ 4669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:44.505856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:44.505874 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:44.505924 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:44.508994 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:44.509833 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:44.509843 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:44.509852 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:44.510139 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:44.510618 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:44.510621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:44.510623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:44.511176 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:44.511375 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:44.511872 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 07:06:44.512250 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":425,"threads_started":1}
I20260927 07:06:44.520426 2474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } }
I20260927 07:06:44.520849 2474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:44.521015 2474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f23f3ef74f845b1bb3460cb756ab709, State: Initialized, Role: FOLLOWER
I20260927 07:06:44.521706 2474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } }
I20260927 07:06:44.521924 2474 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:44.522024 2474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:44.522148 2474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:44.524343 2474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } }
I20260927 07:06:44.524858 2474 leader_election.cc:302] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f23f3ef74f845b1bb3460cb756ab709; no voters:
I20260927 07:06:44.525383 2474 leader_election.cc:288] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:44.525556 2479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:44.526770 2479 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [term 1 LEADER]: Becoming Leader. State: Replica: 3f23f3ef74f845b1bb3460cb756ab709, State: Running, Role: LEADER
I20260927 07:06:44.527328 2479 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } }
I20260927 07:06:44.527760 2474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:44.530441 2480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f23f3ef74f845b1bb3460cb756ab709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } } }
I20260927 07:06:44.530467 2481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f23f3ef74f845b1bb3460cb756ab709. Latest consensus state: current_term: 1 leader_uuid: "3f23f3ef74f845b1bb3460cb756ab709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f23f3ef74f845b1bb3460cb756ab709" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39117 } } }
I20260927 07:06:44.530835 2480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:44.530957 2481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:44.533192 2486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:44.545543 2486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:44.554693 2486 catalog_manager.cc:1347] Generated new cluster ID: 8fab218519a84605ab49745835c645ca
I20260927 07:06:44.554828 2486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:44.579545 2486 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:44.580953 2486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:44.590829 2486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: Generated new TSK 0
I20260927 07:06:44.591768 2486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:44.658598 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:45.028420 2500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:45.028800 2500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:45.029009 2500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:45.036755 2500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:45.036911 2500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:45.037084 2500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:45.047253 2500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:45.048645 2500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:45.050163 2500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:45.063365 2506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:45.063464 2505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:45.063433 2508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:45.064117 2500 server_base.cc:1034] running on GCE node
I20260927 07:06:45.064697 2500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:45.065604 2500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:45.066794 2500 hybrid_clock.cc:648] HybridClock initialized: now 1790492805066741 us; error 50 us; skew 500 ppm
I20260927 07:06:45.067162 2500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:45 dist-test-slave-8vqf krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492805, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:45.070444 2500 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:45.071709 2500 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:45.071796 2500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:45.072095 2500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:45.075150 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "92cdc802443940acab6526c464ff3284"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "490289d5dda1073bbbfc7eb8c83ae0fe"
server_key_iv: "f5c2a9d16462db64a4e2e5a0565ceae8"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.075985 2500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "92cdc802443940acab6526c464ff3284"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "490289d5dda1073bbbfc7eb8c83ae0fe"
server_key_iv: "f5c2a9d16462db64a4e2e5a0565ceae8"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.081337 2500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:45.084803 2514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.086215 2500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:06:45.086382 2500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "92cdc802443940acab6526c464ff3284"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "490289d5dda1073bbbfc7eb8c83ae0fe"
server_key_iv: "f5c2a9d16462db64a4e2e5a0565ceae8"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.086651 2500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:45.111989 2500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:45.116556 2500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:45.116837 2500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:45.117815 2500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:45.119207 2500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:45.119328 2500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.119454 2500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:45.119537 2500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.152779 2500 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41753
I20260927 07:06:45.152864 2626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41753 every 8 connection(s)
I20260927 07:06:45.153679 2500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:45.162177 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2500
I20260927 07:06:45.162398 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:45.163125 30785 external_mini_cluster.cc:1428] Setting key 6328a3fff78b2d1191d65492e210cad4
Sep 27 07:06:45 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492805, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:45.176183 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:45.156373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:45073 (local address 127.30.16.126:39117)
0927 07:06:45.156848 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:45.156853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:45.157932 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 07:06:45.159844 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:45.159850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:45.159961 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:45.160081 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:45.161705 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.162636 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.164475 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.165002 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.165135 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:45.168933 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:45.168939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:45.168942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:45.168977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:45.172636 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:45.173483 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:45.173491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:45.173498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:45.173668 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:45.174088 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:45.174096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:45.174102 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:45.174486 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:45.174691 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:45.175405 (+ 714us) server_negotiation.cc:299] Negotiation successful
0927 07:06:45.175672 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260927 07:06:45.176954 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:45.156925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39117 (local address 127.30.16.65:45073)
0927 07:06:45.157664 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:45.157704 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:45.159542 (+ 1838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:45.160334 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:45.160344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:45.160908 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:45.161503 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:45.161522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.162792 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.162802 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:45.164324 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:45.164337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.165265 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.165273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:45.165518 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:45.166244 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:45.166267 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:45.168671 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:45.172792 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:45.172800 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:45.172816 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:45.173342 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:45.173782 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:45.173786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:45.173788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:45.173978 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:45.174648 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:45.174654 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:45.175012 (+ 358us) client_negotiation.cc:769] Sending connection context
0927 07:06:45.175250 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 07:06:45.175673 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":298,"threads_started":1}
I20260927 07:06:45.178742 2627 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39117
I20260927 07:06:45.179096 2627 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:45.179756 2627 heartbeater.cc:511] Master 127.30.16.126:39117 requested a full tablet report, sending...
I20260927 07:06:45.181885 2439 ts_manager.cc:194] Registered new tserver with Master: 92cdc802443940acab6526c464ff3284 (127.30.16.65:41753)
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:45.183447 2439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:45073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:45.211266 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:45.551489 2633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:45.551776 2633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:45.551887 2633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:45.559324 2633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:45.559448 2633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:45.559568 2633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:45.569547 2633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:45.570878 2633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:45.572165 2633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:45.585115 2638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:45.585193 2641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:45.585142 2639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:45.586282 2633 server_base.cc:1034] running on GCE node
I20260927 07:06:45.586997 2633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:45.588047 2633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:45.589268 2633 hybrid_clock.cc:648] HybridClock initialized: now 1790492805589210 us; error 62 us; skew 500 ppm
I20260927 07:06:45.589615 2633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:45 dist-test-slave-8vqf krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492805, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:45.592900 2633 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:45.594043 2633 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:45.594132 2633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:45.594393 2633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:45.597504 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "7a501a2e5827477e804f2038dbaa529b"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "4e86b0630840f6ff5584976aae540870"
server_key_iv: "3eb5a0802754aba45d717c5c6f653142"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.598264 2633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "7a501a2e5827477e804f2038dbaa529b"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "4e86b0630840f6ff5584976aae540870"
server_key_iv: "3eb5a0802754aba45d717c5c6f653142"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.603442 2633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:06:45.606899 2647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.608304 2633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:45.608490 2633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "7a501a2e5827477e804f2038dbaa529b"
format_stamp: "Formatted at 2026-09-27 07:06:45 on dist-test-slave-8vqf"
server_key: "4e86b0630840f6ff5584976aae540870"
server_key_iv: "3eb5a0802754aba45d717c5c6f653142"
server_key_version: "encryptionkey@0"
I20260927 07:06:45.608701 2633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:45.626827 2633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:45.631387 2633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:45.631662 2633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:45.632714 2633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:45.634124 2633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:45.634208 2633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.634284 2633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:45.634339 2633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:45.668315 2633 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:42677
I20260927 07:06:45.668380 2759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:42677 every 8 connection(s)
I20260927 07:06:45.669243 2633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:45.670985 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2633
I20260927 07:06:45.671176 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:45.671659 30785 external_mini_cluster.cc:1428] Setting key 64ac9a49226adcd57faebd40847e225a
Sep 27 07:06:45 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492805, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:45.691682 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:45.671972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:50005 (local address 127.30.16.126:39117)
0927 07:06:45.672186 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:45.672193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:45.673558 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 07:06:45.675467 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:45.675475 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:45.675563 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:45.675679 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:45.677175 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.678339 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.679976 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.680468 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.680585 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:45.685031 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:45.685037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:45.685041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:45.685075 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:45.687903 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:45.688702 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:45.688707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:45.688710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:45.688844 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:45.689307 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:45.689317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:45.689320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:45.689627 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:45.689798 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:45.690851 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0927 07:06:45.691298 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:06:45.692751 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:45.672579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39117 (local address 127.30.16.66:50005)
0927 07:06:45.673363 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:45.673417 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:45.675202 (+ 1785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:45.675871 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:45.675879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:45.676426 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:45.676997 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:45.677019 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.678505 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.678509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:45.679825 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:45.679836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:45.680703 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:45.680733 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:45.681075 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:45.681868 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:45.681901 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:45.684758 (+ 2857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:45.688059 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:45.688070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:45.688090 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:45.688557 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:45.688976 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:45.688983 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:45.688987 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:45.689183 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:45.689764 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:45.689772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:45.690281 (+ 509us) client_negotiation.cc:769] Sending connection context
0927 07:06:45.690713 (+ 432us) client_negotiation.cc:240] Negotiation successful
0927 07:06:45.691339 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:45.694483 2760 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39117
I20260927 07:06:45.694835 2760 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:45.695515 2760 heartbeater.cc:511] Master 127.30.16.126:39117 requested a full tablet report, sending...
I20260927 07:06:45.697067 2439 ts_manager.cc:194] Registered new tserver with Master: 7a501a2e5827477e804f2038dbaa529b (127.30.16.66:42677)
I20260927 07:06:45.697870 2439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:50005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:45.719058 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:46.046540 2765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:46.046943 2765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:46.047104 2765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:46.054499 2765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:46.054692 2765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:46.054875 2765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:46.065095 2765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:46.066504 2765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:46.068039 2765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:46.081302 2770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:46.081271 2771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:46.081562 2773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:46.082504 2765 server_base.cc:1034] running on GCE node
I20260927 07:06:46.083211 2765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:46.084153 2765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:46.085362 2765 hybrid_clock.cc:648] HybridClock initialized: now 1790492806085310 us; error 40 us; skew 500 ppm
I20260927 07:06:46.085703 2765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:46 dist-test-slave-8vqf krb5kdc[2395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492806, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:46.088973 2765 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:46.090116 2765 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:46.090219 2765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:46.090590 2765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:46.093766 2765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "86017dacf8df4492a729a7aedcb94d51"
format_stamp: "Formatted at 2026-09-27 07:06:46 on dist-test-slave-8vqf"
server_key: "245882e3adac600839328ed2808ccf62"
server_key_iv: "faf645c5aa427a61e8467b9a840c0b1c"
server_key_version: "encryptionkey@0"
I20260927 07:06:46.094537 2765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "86017dacf8df4492a729a7aedcb94d51"
format_stamp: "Formatted at 2026-09-27 07:06:46 on dist-test-slave-8vqf"
server_key: "245882e3adac600839328ed2808ccf62"
server_key_iv: "faf645c5aa427a61e8467b9a840c0b1c"
server_key_version: "encryptionkey@0"
I20260927 07:06:46.099809 2765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:06:46.103530 2779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:46.105048 2765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:46.105223 2765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "86017dacf8df4492a729a7aedcb94d51"
format_stamp: "Formatted at 2026-09-27 07:06:46 on dist-test-slave-8vqf"
server_key: "245882e3adac600839328ed2808ccf62"
server_key_iv: "faf645c5aa427a61e8467b9a840c0b1c"
server_key_version: "encryptionkey@0"
I20260927 07:06:46.105448 2765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:46.132573 2765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:46.137142 2765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:46.137440 2765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:46.138513 2765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:46.140128 2765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:46.140249 2765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:46.140370 2765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:46.140455 2765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:46.173448 2765 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:38639
I20260927 07:06:46.173501 2891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:38639 every 8 connection(s)
I20260927 07:06:46.174340 2765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:46.179033 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2765
I20260927 07:06:46.179268 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:46.179836 30785 external_mini_cluster.cc:1428] Setting key 0e72a8c987864a221318a4f8aaa6e548
I20260927 07:06:46.186705 2627 heartbeater.cc:503] Master 127.30.16.126:39117 was elected leader, sending a full tablet report...
Sep 27 07:06:46 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492806, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:46.196014 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.176940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:47191 (local address 127.30.16.126:39117)
0927 07:06:46.177116 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.177121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.178352 (+ 1231us) server_negotiation.cc:407] Connection header received
0927 07:06:46.180925 (+ 2573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.180931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.181071 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.181186 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:46.183277 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.184021 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.185592 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.186013 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.186119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.189541 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:46.189546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:46.189549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:46.189579 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:46.192462 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.193268 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.193274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.193277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.193463 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.193992 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.193997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.193999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.194426 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:46.194612 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.195525 (+ 913us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.195743 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:06:46.197047 2894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.177441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39117 (local address 127.30.16.67:47191)
0927 07:06:46.178194 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.178233 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.180539 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.181494 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.181506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.182303 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:46.183078 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.183105 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.184150 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.184154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.185452 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.185463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.186205 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.186209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.186399 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.187223 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:46.187245 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:46.189352 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:46.192603 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:46.192611 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:46.192626 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:46.193090 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:46.193607 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:46.193610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:46.193613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:46.193855 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:46.194592 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:46.194601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:46.195088 (+ 487us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.195382 (+ 294us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.195929 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":278,"threads_started":1}
I20260927 07:06:46.198359 2892 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39117
I20260927 07:06:46.198714 2892 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:46.199436 2892 heartbeater.cc:511] Master 127.30.16.126:39117 requested a full tablet report, sending...
I20260927 07:06:46.200811 2439 ts_manager.cc:194] Registered new tserver with Master: 86017dacf8df4492a729a7aedcb94d51 (127.30.16.67:38639)
I20260927 07:06:46.201622 2439 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:47191
I20260927 07:06:46.207952 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:46.225446 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.212775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.30.16.126:39117)
0927 07:06:46.213022 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.213031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.213542 (+ 511us) server_negotiation.cc:407] Connection header received
0927 07:06:46.213752 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.213756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.213821 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.213915 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:46.215420 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.216167 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.217509 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.217983 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.218116 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.220441 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:46.220446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:46.220449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:46.220485 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:46.223146 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.223757 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.223760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.223762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.223875 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.224203 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.224206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.224208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.224514 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:46.224692 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.224931 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.225111 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 07:06:46.229580 2439 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:46.236595 2439 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:46.271739 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.250875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60858 (local address 127.30.16.65:41753)
0927 07:06:46.261505 (+ 10630us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.261515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.263125 (+ 1610us) server_negotiation.cc:407] Connection header received
0927 07:06:46.263265 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.263277 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.263650 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.263893 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:46.265290 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.267274 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.269096 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.270518 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.270722 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.270978 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.271227 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.271386 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10419,"thread_start_us":13252,"threads_started":1}
I20260927 07:06:46.271778 2907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.250524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41753 (local address 127.0.0.1:60858)
0927 07:06:46.262785 (+ 12261us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.262808 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.262970 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.264452 (+ 1482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.264463 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.264514 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:46.265035 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.265051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.267425 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.267430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.268891 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.268910 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.270758 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.270764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.270971 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.271032 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.271181 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.271373 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12110,"thread_start_us":10676,"threads_started":1}
I20260927 07:06:46.276530 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 18cb04d91d464e1681b4fd18999d882b (DEFAULT_TABLE table=test-table [id=8bd402622f7d4a74927c03cfdae61b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:46.277745 2904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.250640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:38639 (local address 127.0.0.1:33998)
0927 07:06:46.267063 (+ 16423us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.267081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.267181 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.269033 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.269040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.269069 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:46.269426 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.269437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.271340 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.271345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.275057 (+ 3712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.275077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.276937 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.276944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.277107 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.277141 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.277233 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.277372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16302,"thread_start_us":11832,"threads_started":1}
I20260927 07:06:46.277796 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.251250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.30.16.67:38639)
0927 07:06:46.268309 (+ 17059us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.268323 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.268367 (+ 44us) server_negotiation.cc:407] Connection header received
0927 07:06:46.268487 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.268498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.268806 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.269038 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:46.269603 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.271188 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.275281 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.276735 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.276884 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.277122 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.277287 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.277462 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16810,"thread_start_us":17989,"threads_started":1}
I20260927 07:06:46.278483 2905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.249939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42677 (local address 127.0.0.1:47898)
0927 07:06:46.261759 (+ 11820us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.261823 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.262093 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.269614 (+ 7521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.269618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.269641 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:46.269955 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.269965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.273233 (+ 3268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.273238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.275243 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.275258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.277672 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.277678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.277857 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.277888 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.277969 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.278100 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11474,"mutex_wait_us":25,"thread_start_us":17310,"threads_started":1}
I20260927 07:06:46.278483 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cb04d91d464e1681b4fd18999d882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:46.279309 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.250599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47898 (local address 127.30.16.66:42677)
0927 07:06:46.268975 (+ 18376us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.268985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.269014 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:06:46.269109 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.269118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.269377 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.269581 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:46.270133 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.273028 (+ 2895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.275804 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.277193 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.277333 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.277527 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.278845 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.279000 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18194,"thread_start_us":19160,"threads_started":1}
I20260927 07:06:46.280930 2827 tablet_service.cc:1467] Processing CreateTablet for tablet 18cb04d91d464e1681b4fd18999d882b (DEFAULT_TABLE table=test-table [id=8bd402622f7d4a74927c03cfdae61b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:46.282307 2827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cb04d91d464e1681b4fd18999d882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:46.283041 2695 tablet_service.cc:1467] Processing CreateTablet for tablet 18cb04d91d464e1681b4fd18999d882b (DEFAULT_TABLE table=test-table [id=8bd402622f7d4a74927c03cfdae61b68]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:46.284379 2695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18cb04d91d464e1681b4fd18999d882b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:46.293244 2910 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Bootstrap starting.
I20260927 07:06:46.296603 2910 tablet_bootstrap.cc:654] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:46.297756 2910 log.cc:824] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:46.299283 2911 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Bootstrap starting.
I20260927 07:06:46.300803 2913 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Bootstrap starting.
I20260927 07:06:46.303992 2911 tablet_bootstrap.cc:654] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:46.304625 2913 tablet_bootstrap.cc:654] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:46.305078 2911 log.cc:824] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:46.305599 2910 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: No bootstrap required, opened a new log
I20260927 07:06:46.305831 2913 log.cc:824] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:46.305971 2910 ts_tablet_manager.cc:1397] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.012s
I20260927 07:06:46.307922 2911 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: No bootstrap required, opened a new log
I20260927 07:06:46.308173 2911 ts_tablet_manager.cc:1397] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 07:06:46.309043 2913 tablet_bootstrap.cc:492] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: No bootstrap required, opened a new log
I20260927 07:06:46.309391 2913 ts_tablet_manager.cc:1397] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 07:06:46.316411 2911 raft_consensus.cc:348] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.316828 2911 raft_consensus.cc:374] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:46.316946 2911 raft_consensus.cc:729] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92cdc802443940acab6526c464ff3284, State: Initialized, Role: FOLLOWER
I20260927 07:06:46.316823 2910 raft_consensus.cc:348] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.317453 2910 raft_consensus.cc:374] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:46.317564 2910 raft_consensus.cc:729] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86017dacf8df4492a729a7aedcb94d51, State: Initialized, Role: FOLLOWER
I20260927 07:06:46.317611 2911 consensus_queue.cc:260] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.318164 2910 consensus_queue.cc:260] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.319563 2911 ts_tablet_manager.cc:1428] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260927 07:06:46.321481 2913 raft_consensus.cc:348] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.322057 2913 raft_consensus.cc:374] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:46.322170 2913 raft_consensus.cc:729] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a501a2e5827477e804f2038dbaa529b, State: Initialized, Role: FOLLOWER
I20260927 07:06:46.323231 2913 consensus_queue.cc:260] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.324548 2760 heartbeater.cc:503] Master 127.30.16.126:39117 was elected leader, sending a full tablet report...
I20260927 07:06:46.325367 2913 ts_tablet_manager.cc:1428] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
I20260927 07:06:46.325825 2892 heartbeater.cc:503] Master 127.30.16.126:39117 was elected leader, sending a full tablet report...
I20260927 07:06:46.327294 2910 ts_tablet_manager.cc:1428] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
W20260927 07:06:46.406118 2628 tablet.cc:2369] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:46.421211 2761 tablet.cc:2369] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:06:46.425933 2893 tablet.cc:2369] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:06:46.435153 2916 raft_consensus.cc:482] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:46.435461 2916 raft_consensus.cc:504] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.437179 2916 leader_election.cc:288] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a501a2e5827477e804f2038dbaa529b (127.30.16.66:42677), 86017dacf8df4492a729a7aedcb94d51 (127.30.16.67:38639)
I20260927 07:06:46.444599 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.437622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:37585 (local address 127.30.16.66:42677)
0927 07:06:46.438178 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.438185 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.438944 (+ 759us) server_negotiation.cc:407] Connection header received
0927 07:06:46.439108 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.439111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.439169 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.439314 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:46.439965 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.440701 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.442849 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.443608 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.443789 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.443864 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.444192 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.444318 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260927 07:06:46.444595 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.437628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60719 (local address 127.30.16.67:38639)
0927 07:06:46.437972 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.437982 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.438736 (+ 754us) server_negotiation.cc:407] Connection header received
0927 07:06:46.439051 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.439057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.439134 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.439200 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:06:46.439898 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.440719 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.442794 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.443563 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.443750 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.443830 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.444180 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.444320 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 07:06:46.444783 2919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.437671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42677 (local address 127.30.16.65:37585)
0927 07:06:46.438823 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.438837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.438970 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.439322 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.439330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.439357 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:46.439754 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.439764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.440834 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.440842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.442706 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.442717 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.443831 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.443838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.444043 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.444065 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.444158 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.444308 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":205,"thread_start_us":411,"threads_started":1}
I20260927 07:06:46.444878 2920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.437543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:38639 (local address 127.30.16.65:60719)
0927 07:06:46.438494 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:46.438518 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:46.438848 (+ 330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:46.439242 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:46.439249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:46.439275 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:06:46.439727 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.439737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.440845 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.440853 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.442637 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:46.442649 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.443796 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.443804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:46.444029 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.444053 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 07:06:46.444153 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 07:06:46.444312 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":69,"thread_start_us":804,"threads_started":1}
I20260927 07:06:46.445844 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cb04d91d464e1681b4fd18999d882b" candidate_uuid: "92cdc802443940acab6526c464ff3284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a501a2e5827477e804f2038dbaa529b" is_pre_election: true
I20260927 07:06:46.445935 2847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cb04d91d464e1681b4fd18999d882b" candidate_uuid: "92cdc802443940acab6526c464ff3284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86017dacf8df4492a729a7aedcb94d51" is_pre_election: true
I20260927 07:06:46.446368 2715 raft_consensus.cc:2454] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cdc802443940acab6526c464ff3284 in term 0.
I20260927 07:06:46.446408 2847 raft_consensus.cc:2454] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92cdc802443940acab6526c464ff3284 in term 0.
I20260927 07:06:46.447055 2517 leader_election.cc:302] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a501a2e5827477e804f2038dbaa529b, 86017dacf8df4492a729a7aedcb94d51, 92cdc802443940acab6526c464ff3284; no voters:
I20260927 07:06:46.447465 2916 raft_consensus.cc:2781] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:46.447600 2916 raft_consensus.cc:482] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:46.447712 2916 raft_consensus.cc:3037] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:46.449963 2916 raft_consensus.cc:504] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.450723 2916 leader_election.cc:288] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [CANDIDATE]: Term 1 election: Requested vote from peers 7a501a2e5827477e804f2038dbaa529b (127.30.16.66:42677), 86017dacf8df4492a729a7aedcb94d51 (127.30.16.67:38639)
I20260927 07:06:46.451161 2715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cb04d91d464e1681b4fd18999d882b" candidate_uuid: "92cdc802443940acab6526c464ff3284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a501a2e5827477e804f2038dbaa529b"
I20260927 07:06:46.451165 2847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18cb04d91d464e1681b4fd18999d882b" candidate_uuid: "92cdc802443940acab6526c464ff3284" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86017dacf8df4492a729a7aedcb94d51"
I20260927 07:06:46.451376 2715 raft_consensus.cc:3037] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:46.451391 2847 raft_consensus.cc:3037] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:46.453528 2715 raft_consensus.cc:2454] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cdc802443940acab6526c464ff3284 in term 1.
I20260927 07:06:46.453534 2847 raft_consensus.cc:2454] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92cdc802443940acab6526c464ff3284 in term 1.
I20260927 07:06:46.453915 2517 leader_election.cc:302] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7a501a2e5827477e804f2038dbaa529b, 86017dacf8df4492a729a7aedcb94d51, 92cdc802443940acab6526c464ff3284; no voters:
I20260927 07:06:46.454190 2916 raft_consensus.cc:2781] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:46.454679 2916 raft_consensus.cc:686] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 1 LEADER]: Becoming Leader. State: Replica: 92cdc802443940acab6526c464ff3284, State: Running, Role: LEADER
I20260927 07:06:46.455268 2916 consensus_queue.cc:237] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } }
I20260927 07:06:46.459704 2439 catalog_manager.cc:5495] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92cdc802443940acab6526c464ff3284 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "92cdc802443940acab6526c464ff3284" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92cdc802443940acab6526c464ff3284" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41753 } health_report { overall_health: HEALTHY } } }
I20260927 07:06:46.510041 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.504304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60860 (local address 127.30.16.65:41753)
0927 07:06:46.504541 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.504546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.504564 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:06:46.504652 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.504656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.504732 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.504875 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:06:46.505208 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.506014 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.507643 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.508052 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.508148 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.508766 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:06:46.509335 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:06:46.509556 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.509668 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.509752 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 07:06:46.518704 2847 raft_consensus.cc:1261] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 1 FOLLOWER]: Refusing update from remote peer 92cdc802443940acab6526c464ff3284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:46.518786 2715 raft_consensus.cc:1261] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 1 FOLLOWER]: Refusing update from remote peer 92cdc802443940acab6526c464ff3284: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:46.519757 2921 consensus_queue.cc:1035] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a501a2e5827477e804f2038dbaa529b" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:46.520123 2916 consensus_queue.cc:1035] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86017dacf8df4492a729a7aedcb94d51" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 38639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:46.549490 2438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:46.549800 2438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47000:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:46.553812 2438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f23f3ef74f845b1bb3460cb756ab709: Sending DeleteTablet for 3 replicas of tablet 18cb04d91d464e1681b4fd18999d882b
I20260927 07:06:46.555143 2827 tablet_service.cc:1514] Processing DeleteTablet for tablet 18cb04d91d464e1681b4fd18999d882b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:46 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33998
I20260927 07:06:46.555198 2695 tablet_service.cc:1514] Processing DeleteTablet for tablet 18cb04d91d464e1681b4fd18999d882b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:46 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47898
I20260927 07:06:46.557487 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 18cb04d91d464e1681b4fd18999d882b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:46 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:60858
I20260927 07:06:46.558395 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2500
I20260927 07:06:46.585001 2936 tablet_replica.cc:330] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: stopping tablet replica
I20260927 07:06:46.587618 2936 raft_consensus.cc:2229] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:46.590157 2936 raft_consensus.cc:2258] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:46.595657 2935 tablet_replica.cc:330] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: stopping tablet replica
I20260927 07:06:46.596755 2934 tablet_replica.cc:330] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: stopping tablet replica
I20260927 07:06:46.596762 2935 raft_consensus.cc:2229] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:46.597239 2934 raft_consensus.cc:2229] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:46.597350 2935 raft_consensus.cc:2258] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:46.597833 2934 raft_consensus.cc:2258] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:46.599661 2935 ts_tablet_manager.cc:1905] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:46.599965 2936 ts_tablet_manager.cc:1905] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:46.600407 2934 ts_tablet_manager.cc:1905] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 07:06:46 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:46.607030 2935 ts_tablet_manager.cc:1918] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:46.607213 2935 log.cc:1196] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/18cb04d91d464e1681b4fd18999d882b
I20260927 07:06:46.607770 2935 ts_tablet_manager.cc:1939] T 18cb04d91d464e1681b4fd18999d882b P 7a501a2e5827477e804f2038dbaa529b: Deleting consensus metadata
I20260927 07:06:46.609901 2425 catalog_manager.cc:4841] TS 7a501a2e5827477e804f2038dbaa529b (127.30.16.66:42677): tablet 18cb04d91d464e1681b4fd18999d882b (table test-table [id=8bd402622f7d4a74927c03cfdae61b68]) successfully deleted
I20260927 07:06:46.611344 2908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.563143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.30.16.65:41753)
0927 07:06:46.563386 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.563396 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.583660 (+ 20264us) server_negotiation.cc:407] Connection header received
0927 07:06:46.585338 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.585347 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.585461 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.585570 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:46.590420 (+ 4850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.592193 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.596116 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.596829 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.596972 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.602719 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:46.602734 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:46.602754 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:46.602807 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:46.606998 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.608100 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.608125 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.608129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.608366 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.608813 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.608818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.608823 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.609326 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:46.609617 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.610651 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.610923 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 07:06:46.612267 2934 ts_tablet_manager.cc:1918] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:46.612529 2934 log.cc:1196] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/18cb04d91d464e1681b4fd18999d882b
I20260927 07:06:46.613095 2622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:46.613157 2934 ts_tablet_manager.cc:1939] T 18cb04d91d464e1681b4fd18999d882b P 86017dacf8df4492a729a7aedcb94d51: Deleting consensus metadata
I20260927 07:06:46.615212 2424 catalog_manager.cc:4841] TS 86017dacf8df4492a729a7aedcb94d51 (127.30.16.67:38639): tablet 18cb04d91d464e1681b4fd18999d882b (table test-table [id=8bd402622f7d4a74927c03cfdae61b68]) successfully deleted
I20260927 07:06:46.723416 2936 ts_tablet_manager.cc:1918] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:46.723677 2936 log.cc:1196] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/18cb04d91d464e1681b4fd18999d882b
I20260927 07:06:46.724428 2936 ts_tablet_manager.cc:1939] T 18cb04d91d464e1681b4fd18999d882b P 92cdc802443940acab6526c464ff3284: Deleting consensus metadata
I20260927 07:06:46.726425 2426 catalog_manager.cc:4841] TS 92cdc802443940acab6526c464ff3284 (127.30.16.65:41753): tablet 18cb04d91d464e1681b4fd18999d882b (table test-table [id=8bd402622f7d4a74927c03cfdae61b68]) successfully deleted
I20260927 07:06:46.726744 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2500
I20260927 07:06:46.746654 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2633
Sep 27 07:06:46 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:46.761456 2906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.747378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.30.16.66:42677)
0927 07:06:46.747549 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.747554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.747577 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:06:46.747742 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.747746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.747812 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.747953 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:46.749191 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.750000 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.751420 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.751817 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.751933 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.755969 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:46.755984 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:46.756000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:46.756034 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:46.758812 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.759450 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.759460 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.759463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.759586 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.759942 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.759947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.759951 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.760314 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:46.760602 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.760888 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.761109 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:06:46.762099 2755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:46.852535 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2633
I20260927 07:06:46.869170 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2765
Sep 27 07:06:46 dist-test-slave-8vqf krb5kdc[2395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:46.884467 2909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:46.870009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.30.16.67:38639)
0927 07:06:46.870206 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:46.870212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:46.870234 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:06:46.870382 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:46.870388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:46.870466 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:46.870597 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:46.871858 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.872581 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.873867 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:46.874262 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:46.874399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:46.878803 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:46.878822 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:46.878840 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:46.878887 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:46.881643 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.882305 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.882313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.882315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.882449 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:46.882911 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:46.882924 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:46.882926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:46.883270 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:46.883576 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:46.883906 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 07:06:46.884135 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:06:46.885104 2887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:46.973968 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2765
I20260927 07:06:46.990370 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2408
I20260927 07:06:46.991554 2469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:47.100940 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2408
2026-09-27T07:06:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4407 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T07:06:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:47Z Disabled control of system clock
I20260927 07:06:47.173096 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:35999
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:35999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:06:47.509931 2945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:47.510238 2945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:47.510403 2945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:47.518024 2945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:47.518151 2945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:47.518221 2945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:47.518308 2945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:47.518383 2945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:47.528272 2945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:35999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:35999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:47.529677 2945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:47.531870 2945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:47.544313 2950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:47.544379 2951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:47.544348 2953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:47.545186 2945 server_base.cc:1034] running on GCE node
I20260927 07:06:47.545974 2945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:47.547468 2945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:47.548671 2945 hybrid_clock.cc:648] HybridClock initialized: now 1790492807548628 us; error 31 us; skew 500 ppm
I20260927 07:06:47.549037 2945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:47.556190 2945 webserver.cc:460] Webserver started at http://127.30.16.126:38339/ using document root <none> and password file <none>
I20260927 07:06:47.556982 2945 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:47.557101 2945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:47.557441 2945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:47.560664 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "1d84616321744316858eddbeaae6c029"
format_stamp: "Formatted at 2026-09-27 07:06:47 on dist-test-slave-8vqf"
server_key: "42f3b2e5061d8b2dd81132552cb72a1f"
server_key_iv: "8a771f5774faf1c7f5d623f54b11b36d"
server_key_version: "encryptionkey@0"
I20260927 07:06:47.561496 2945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "1d84616321744316858eddbeaae6c029"
format_stamp: "Formatted at 2026-09-27 07:06:47 on dist-test-slave-8vqf"
server_key: "42f3b2e5061d8b2dd81132552cb72a1f"
server_key_iv: "8a771f5774faf1c7f5d623f54b11b36d"
server_key_version: "encryptionkey@0"
I20260927 07:06:47.566939 2945 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 07:06:47.570701 2959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:47.572176 2945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:47.572402 2945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "1d84616321744316858eddbeaae6c029"
format_stamp: "Formatted at 2026-09-27 07:06:47 on dist-test-slave-8vqf"
server_key: "42f3b2e5061d8b2dd81132552cb72a1f"
server_key_iv: "8a771f5774faf1c7f5d623f54b11b36d"
server_key_version: "encryptionkey@0"
I20260927 07:06:47.572654 2945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:47.578702 2945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:47.579468 2945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:47.579756 2945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:47.598017 2945 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:35999
I20260927 07:06:47.598071 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:35999 every 8 connection(s)
I20260927 07:06:47.599509 2945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:47.603322 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 2945
I20260927 07:06:47.603348 3011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:47.603528 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:47.604202 30785 external_mini_cluster.cc:1428] Setting key 68d998cf2c37a107f23b187f069d0035
I20260927 07:06:47.612910 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:47.606121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.30.16.126:35999)
0927 07:06:47.607199 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:47.607209 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:47.607254 (+ 45us) server_negotiation.cc:407] Connection header received
0927 07:06:47.609245 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:47.609269 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:47.609714 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:47.610067 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:47.611238 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:47.611245 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:47.611258 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:47.611273 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:47.611338 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:47.611355 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:47.611515 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:47.611641 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 07:06:47.611972 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":418,"threads_started":1}
I20260927 07:06:47.616780 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: Bootstrap starting.
I20260927 07:06:47.619894 3011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:47.620895 3011 log.cc:824] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:47.623973 3011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: No bootstrap required, opened a new log
I20260927 07:06:47.631271 3011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } }
I20260927 07:06:47.631657 3011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:47.631759 3011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d84616321744316858eddbeaae6c029, State: Initialized, Role: FOLLOWER
I20260927 07:06:47.632436 3011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } }
I20260927 07:06:47.632681 3011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:47.632820 3011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:47.632975 3011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:47.635129 3011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } }
I20260927 07:06:47.635576 3011 leader_election.cc:302] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d84616321744316858eddbeaae6c029; no voters:
I20260927 07:06:47.636112 3011 leader_election.cc:288] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:47.636267 3016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:47.636955 3016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [term 1 LEADER]: Becoming Leader. State: Replica: 1d84616321744316858eddbeaae6c029, State: Running, Role: LEADER
I20260927 07:06:47.637490 3016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } }
I20260927 07:06:47.638156 3011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:47.640213 3017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d84616321744316858eddbeaae6c029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } } }
I20260927 07:06:47.640664 3017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:47.641072 3018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d84616321744316858eddbeaae6c029. Latest consensus state: current_term: 1 leader_uuid: "1d84616321744316858eddbeaae6c029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d84616321744316858eddbeaae6c029" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 35999 } } }
I20260927 07:06:47.641418 3018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:47.642740 3021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:47.648224 3021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:47.657593 3021 catalog_manager.cc:1347] Generated new cluster ID: acc2bd6ed6eb4b1db059df5f33a0fe69
I20260927 07:06:47.657714 3021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:47.675441 3021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:47.675912 3021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:47.696913 3021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: Generated new TSK 0
I20260927 07:06:47.697837 3021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:06:47.715531 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--builtin_ntp_servers=127.30.16.84:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 07:06:48.066705 3035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:48.067050 3035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:48.067205 3035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:48.074708 3035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:48.074817 3035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:48.074990 3035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:48.084533 3035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:48.085865 3035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:48.087770 3035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:48.102121 3040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.102099 3041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.102150 3043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:48.103125 3035 server_base.cc:1034] running on GCE node
I20260927 07:06:48.103760 3035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:48.104763 3035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:48.106019 3035 hybrid_clock.cc:648] HybridClock initialized: now 1790492808105996 us; error 75 us; skew 500 ppm
I20260927 07:06:48.106353 3035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:48.108901 3035 webserver.cc:460] Webserver started at http://127.30.16.65:35181/ using document root <none> and password file <none>
I20260927 07:06:48.109629 3035 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:48.109756 3035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:48.110095 3035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:48.113420 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "45874e1761fa4033aa6e89dfe80555df"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "864cfd73feb5e69ad855d96614c97b31"
server_key_iv: "ac1c9a99c4f17844e39941b2b9dd0e15"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.114238 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "45874e1761fa4033aa6e89dfe80555df"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "864cfd73feb5e69ad855d96614c97b31"
server_key_iv: "ac1c9a99c4f17844e39941b2b9dd0e15"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.119797 3035 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:06:48.123427 3049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.124814 3035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:48.125002 3035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "45874e1761fa4033aa6e89dfe80555df"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "864cfd73feb5e69ad855d96614c97b31"
server_key_iv: "ac1c9a99c4f17844e39941b2b9dd0e15"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.125233 3035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:48.131070 3035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:48.131862 3035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:48.132176 3035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:48.133126 3035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:48.134511 3035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:48.134652 3035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.134781 3035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:48.134863 3035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.167975 3035 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:38539
I20260927 07:06:48.168036 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:38539 every 8 connection(s)
I20260927 07:06:48.169397 3035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:48.178181 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:48.172134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:47905 (local address 127.30.16.126:35999)
0927 07:06:48.172596 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:48.172602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:48.173869 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 07:06:48.175639 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:48.175643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:48.175698 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:48.175843 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:48.177457 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:48.177461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:48.177464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:48.177481 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:48.177523 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:48.177534 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:48.177691 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:48.177817 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 07:06:48.177872 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260927 07:06:48.178638 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3035
I20260927 07:06:48.178814 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:48.179263 3165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:48.172800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:35999 (local address 127.30.16.65:47905)
0927 07:06:48.173709 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:48.173755 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:48.175379 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:48.176089 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:48.176098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:48.176341 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:48.177146 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:48.177148 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:48.177188 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:48.177280 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:48.177632 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:48.177640 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:48.177647 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:06:48.177885 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 07:06:48.178117 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":259,"threads_started":1}
I20260927 07:06:48.179519 30785 external_mini_cluster.cc:1428] Setting key ac66d759d49fccb0f27ff34c3ee3511b
I20260927 07:06:48.181092 3162 heartbeater.cc:347] Connected to a master server at 127.30.16.126:35999
I20260927 07:06:48.181407 3162 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:48.182096 3162 heartbeater.cc:511] Master 127.30.16.126:35999 requested a full tablet report, sending...
I20260927 07:06:48.182519 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--builtin_ntp_servers=127.30.16.84:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 07:06:48.184638 2976 ts_manager.cc:194] Registered new tserver with Master: 45874e1761fa4033aa6e89dfe80555df (127.30.16.65:38539)
W20260927 07:06:48.516093 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:48.516433 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:48.516578 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:48.523919 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:48.524091 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:48.524273 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:48.534101 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:48.536031 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:48.538671 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:48.553840 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.553819 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.554021 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:48.554665 3166 server_base.cc:1034] running on GCE node
I20260927 07:06:48.555358 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:48.556438 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:48.557670 3166 hybrid_clock.cc:648] HybridClock initialized: now 1790492808557626 us; error 60 us; skew 500 ppm
I20260927 07:06:48.558012 3166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:48.560650 3166 webserver.cc:460] Webserver started at http://127.30.16.66:42007/ using document root <none> and password file <none>
I20260927 07:06:48.561371 3166 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:48.561501 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:48.561836 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:48.564786 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "db80a9dcb7b0411799f6bcbb0022eea3"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "7d0f928825ce5899791bd3bf59f23e7e"
server_key_iv: "adccb463ffc0efe6bfe642d609d7dd6e"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.565589 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "db80a9dcb7b0411799f6bcbb0022eea3"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "7d0f928825ce5899791bd3bf59f23e7e"
server_key_iv: "adccb463ffc0efe6bfe642d609d7dd6e"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.570902 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 07:06:48.574407 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.575865 3166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:48.576097 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "db80a9dcb7b0411799f6bcbb0022eea3"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "7d0f928825ce5899791bd3bf59f23e7e"
server_key_iv: "adccb463ffc0efe6bfe642d609d7dd6e"
server_key_version: "encryptionkey@0"
I20260927 07:06:48.576339 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:48.582029 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:48.582895 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:48.583160 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:48.584152 3166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:48.585520 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:48.585635 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.585763 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:48.585847 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:48.620448 3166 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:39013
I20260927 07:06:48.620498 3292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:39013 every 8 connection(s)
I20260927 07:06:48.621760 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:48.623765 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3166
I20260927 07:06:48.623935 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:48.624413 30785 external_mini_cluster.cc:1428] Setting key 5725b8a20fe472b35331f99573d81454
I20260927 07:06:48.627385 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--builtin_ntp_servers=127.30.16.84:40991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 07:06:48.630709 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:48.624495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:36525 (local address 127.30.16.126:35999)
0927 07:06:48.624697 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:48.624704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:48.626400 (+ 1696us) server_negotiation.cc:407] Connection header received
0927 07:06:48.628193 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:48.628202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:48.628283 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:48.628436 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:48.629858 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:48.629867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:48.629874 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:48.629897 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:48.629950 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:48.629969 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:48.630084 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:48.630283 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 07:06:48.630350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 07:06:48.631534 3295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:48.625048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:35999 (local address 127.30.16.66:36525)
0927 07:06:48.626206 (+ 1158us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:48.626267 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:48.627939 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:48.628724 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:48.628733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:48.628827 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:48.629590 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:48.629593 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:48.629631 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:48.629716 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:48.630062 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:48.630069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:48.630088 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 07:06:48.630322 (+ 234us) client_negotiation.cc:240] Negotiation successful
0927 07:06:48.630529 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260927 07:06:48.633122 3293 heartbeater.cc:347] Connected to a master server at 127.30.16.126:35999
I20260927 07:06:48.633484 3293 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:48.634061 3293 heartbeater.cc:511] Master 127.30.16.126:35999 requested a full tablet report, sending...
I20260927 07:06:48.635514 2976 ts_manager.cc:194] Registered new tserver with Master: db80a9dcb7b0411799f6bcbb0022eea3 (127.30.16.66:39013)
W20260927 07:06:48.957433 3296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:48.957763 3296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:48.957913 3296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:48.965463 3296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:48.965619 3296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:48.965799 3296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:48.975086 3296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:35999
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:48.976501 3296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:48.978392 3296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:48.992213 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.992151 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:48.992363 3304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:48.993983 3296 server_base.cc:1034] running on GCE node
I20260927 07:06:48.994671 3296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:48.995671 3296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:48.996912 3296 hybrid_clock.cc:648] HybridClock initialized: now 1790492808996845 us; error 63 us; skew 500 ppm
I20260927 07:06:48.997252 3296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:48.999742 3296 webserver.cc:460] Webserver started at http://127.30.16.67:33073/ using document root <none> and password file <none>
I20260927 07:06:49.000484 3296 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:49.000613 3296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:49.000954 3296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:49.004092 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "5f3a4b71a5c244a7862d6b7be004b660"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "040e3cb2ad10c2630a4090e8a80aa164"
server_key_iv: "e110d56131eddbf7e361ef2ce88d36dd"
server_key_version: "encryptionkey@0"
I20260927 07:06:49.004933 3296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3a4b71a5c244a7862d6b7be004b660"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "040e3cb2ad10c2630a4090e8a80aa164"
server_key_iv: "e110d56131eddbf7e361ef2ce88d36dd"
server_key_version: "encryptionkey@0"
I20260927 07:06:49.010730 3296 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260927 07:06:49.014403 3310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:49.015982 3296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:49.016185 3296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "5f3a4b71a5c244a7862d6b7be004b660"
format_stamp: "Formatted at 2026-09-27 07:06:48 on dist-test-slave-8vqf"
server_key: "040e3cb2ad10c2630a4090e8a80aa164"
server_key_iv: "e110d56131eddbf7e361ef2ce88d36dd"
server_key_version: "encryptionkey@0"
I20260927 07:06:49.016451 3296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:49.022771 3296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:49.023672 3296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:49.023959 3296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:49.024945 3296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:49.026515 3296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:49.026677 3296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 07:06:49.026808 3296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:49.026890 3296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:49.060459 3296 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:43917
I20260927 07:06:49.060492 3422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:43917 every 8 connection(s)
I20260927 07:06:49.061789 3296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:49.069573 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3296
I20260927 07:06:49.069758 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:49.070011 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.064407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:43971 (local address 127.30.16.126:35999)
0927 07:06:49.064617 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.064624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.066117 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 07:06:49.067823 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.067828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.067891 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.067988 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.069234 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.069240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.069243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.069259 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.069289 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.069300 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.069438 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.069651 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.069722 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:06:49.070329 30785 external_mini_cluster.cc:1428] Setting key 2e241698873ae849206abac282208b4e
I20260927 07:06:49.071202 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.065029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:35999 (local address 127.30.16.67:43971)
0927 07:06:49.065958 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.065996 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.067567 (+ 1571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.068198 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.068207 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.068303 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.068939 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.068941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.068985 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.069068 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.069409 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.069438 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.069449 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.069731 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.069993 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":255,"threads_started":1}
I20260927 07:06:49.072449 3423 heartbeater.cc:347] Connected to a master server at 127.30.16.126:35999
I20260927 07:06:49.072764 3423 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:49.073446 3423 heartbeater.cc:511] Master 127.30.16.126:35999 requested a full tablet report, sending...
I20260927 07:06:49.074767 2976 ts_manager.cc:194] Registered new tserver with Master: 5f3a4b71a5c244a7862d6b7be004b660 (127.30.16.67:43917)
I20260927 07:06:49.085596 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:06:49.092540 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.089915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42910 (local address 127.30.16.126:35999)
0927 07:06:49.090099 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.090108 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.090536 (+ 428us) server_negotiation.cc:407] Connection header received
0927 07:06:49.090759 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.090764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.090828 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.090936 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.091868 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.091873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.091878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.091895 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.091939 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.091951 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.092071 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.092178 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.092252 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 07:06:49.096385 2976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:06:49.103129 2976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:06:49.120878 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.116140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49940 (local address 127.30.16.65:38539)
0927 07:06:49.116900 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.116910 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.117602 (+ 692us) server_negotiation.cc:407] Connection header received
0927 07:06:49.117681 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.117688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.117956 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.118109 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.120013 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.120020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.120032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.120056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.120103 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.120119 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.120286 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.120430 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.120539 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":429,"threads_started":1}
I20260927 07:06:49.121081 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.115541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:49940)
0927 07:06:49.117360 (+ 1819us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.117391 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.117538 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.118740 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.118747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.118847 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.119715 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.119720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.119756 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.119837 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.120226 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.120236 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.120244 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.120466 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.120595 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1687,"thread_start_us":1548,"threads_started":1}
I20260927 07:06:49.121915 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.115090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.30.16.66:39013)
0927 07:06:49.116323 (+ 1233us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.116336 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.117497 (+ 1161us) server_negotiation.cc:407] Connection header received
0927 07:06:49.117741 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.117751 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.117990 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.118111 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.120757 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.120768 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.120784 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.120803 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.120998 (+ 195us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.121031 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.121172 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.121464 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.121569 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":1283,"threads_started":1}
I20260927 07:06:49.123283 3435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.114939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39013 (local address 127.0.0.1:56472)
0927 07:06:49.117338 (+ 2399us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.117392 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.117582 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.119208 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.119215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.119242 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.120479 (+ 1237us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.120482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.120498 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.120573 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.121268 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.121275 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.121280 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.122694 (+ 1414us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.122762 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2135,"thread_start_us":1939,"threads_started":1}
I20260927 07:06:49.124354 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.118074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.30.16.67:43917)
0927 07:06:49.119990 (+ 1916us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.120000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.120278 (+ 278us) server_negotiation.cc:407] Connection header received
0927 07:06:49.120358 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.120368 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.120677 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.120817 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.123530 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.123542 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.123556 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.123574 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.123658 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.123681 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.123843 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.123901 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.123988 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":1535,"threads_started":1}
I20260927 07:06:49.124372 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.118483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:43917 (local address 127.0.0.1:55162)
0927 07:06:49.120136 (+ 1653us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.120150 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.120262 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.121527 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.121531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.121553 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.123261 (+ 1708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.123266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.123288 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.123331 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.123775 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.123783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.123789 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.123919 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.123966 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"thread_start_us":871,"threads_started":1}
I20260927 07:06:49.125838 3097 tablet_service.cc:1467] Processing CreateTablet for tablet b979a48ab842485e86d4ef89234613ee (DEFAULT_TABLE table=test-table [id=8a88e0984f494c8ebf8e3b2b0e1480fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:49.126540 3228 tablet_service.cc:1467] Processing CreateTablet for tablet b979a48ab842485e86d4ef89234613ee (DEFAULT_TABLE table=test-table [id=8a88e0984f494c8ebf8e3b2b0e1480fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:49.127385 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979a48ab842485e86d4ef89234613ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:49.127990 3228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979a48ab842485e86d4ef89234613ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:49.127985 3358 tablet_service.cc:1467] Processing CreateTablet for tablet b979a48ab842485e86d4ef89234613ee (DEFAULT_TABLE table=test-table [id=8a88e0984f494c8ebf8e3b2b0e1480fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:06:49.129258 3358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b979a48ab842485e86d4ef89234613ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:49.140471 3441 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Bootstrap starting.
I20260927 07:06:49.142634 3442 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Bootstrap starting.
I20260927 07:06:49.144558 3441 tablet_bootstrap.cc:654] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:49.145941 3441 log.cc:824] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:49.146232 3442 tablet_bootstrap.cc:654] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:49.147553 3442 log.cc:824] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:49.149708 3441 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: No bootstrap required, opened a new log
I20260927 07:06:49.150032 3443 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Bootstrap starting.
I20260927 07:06:49.150064 3441 ts_tablet_manager.cc:1397] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260927 07:06:49.153124 3443 tablet_bootstrap.cc:654] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:49.154488 3443 log.cc:824] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:49.156278 3442 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: No bootstrap required, opened a new log
I20260927 07:06:49.156574 3442 ts_tablet_manager.cc:1397] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 07:06:49.157686 3443 tablet_bootstrap.cc:492] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: No bootstrap required, opened a new log
I20260927 07:06:49.157940 3443 ts_tablet_manager.cc:1397] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260927 07:06:49.161417 3441 raft_consensus.cc:348] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.161880 3441 raft_consensus.cc:374] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:49.161972 3441 raft_consensus.cc:729] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db80a9dcb7b0411799f6bcbb0022eea3, State: Initialized, Role: FOLLOWER
I20260927 07:06:49.162703 3441 consensus_queue.cc:260] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.163865 3293 heartbeater.cc:503] Master 127.30.16.126:35999 was elected leader, sending a full tablet report...
I20260927 07:06:49.166158 3441 ts_tablet_manager.cc:1428] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 07:06:49.166016 3443 raft_consensus.cc:348] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.166466 3443 raft_consensus.cc:374] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:49.166579 3443 raft_consensus.cc:729] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3a4b71a5c244a7862d6b7be004b660, State: Initialized, Role: FOLLOWER
I20260927 07:06:49.167042 3442 raft_consensus.cc:348] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.167439 3442 raft_consensus.cc:374] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:49.167316 3443 consensus_queue.cc:260] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.167527 3442 raft_consensus.cc:729] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45874e1761fa4033aa6e89dfe80555df, State: Initialized, Role: FOLLOWER
I20260927 07:06:49.173478 3442 consensus_queue.cc:260] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.182765 3162 heartbeater.cc:503] Master 127.30.16.126:35999 was elected leader, sending a full tablet report...
I20260927 07:06:49.183055 3423 heartbeater.cc:503] Master 127.30.16.126:35999 was elected leader, sending a full tablet report...
I20260927 07:06:49.183662 3442 ts_tablet_manager.cc:1428] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Time spent starting tablet: real 0.027s user 0.018s sys 0.004s
I20260927 07:06:49.184415 3448 raft_consensus.cc:482] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:06:49.184737 3448 raft_consensus.cc:504] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.185822 3443 ts_tablet_manager.cc:1428] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Time spent starting tablet: real 0.028s user 0.008s sys 0.011s
I20260927 07:06:49.187855 3448 leader_election.cc:288] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db80a9dcb7b0411799f6bcbb0022eea3 (127.30.16.66:39013), 45874e1761fa4033aa6e89dfe80555df (127.30.16.65:38539)
I20260927 07:06:49.191305 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.188860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:51145 (local address 127.30.16.66:39013)
0927 07:06:49.189036 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.189044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.189063 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:06:49.189123 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.189127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.189189 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.189289 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.190498 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.190506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.190512 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.190534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.190596 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.190617 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.190744 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.190954 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.191013 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:06:49.191533 3425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.188155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39013 (local address 127.30.16.67:51145)
0927 07:06:49.188323 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.188346 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.188442 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.189464 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.189472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.189504 (+ 32us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.190223 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.190229 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.190251 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.190308 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.190798 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.190807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.190815 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.191119 (+ 304us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.191194 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 07:06:49.192551 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979a48ab842485e86d4ef89234613ee" candidate_uuid: "5f3a4b71a5c244a7862d6b7be004b660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" is_pre_election: true
I20260927 07:06:49.192783 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.189175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:38565 (local address 127.30.16.65:38539)
0927 07:06:49.189335 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.189340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.190306 (+ 966us) server_negotiation.cc:407] Connection header received
0927 07:06:49.190524 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.190529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.190609 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.190679 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.192036 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.192042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.192046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.192066 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.192103 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.192118 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.192207 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.192442 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.192496 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:06:49.193140 3248 raft_consensus.cc:2454] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3a4b71a5c244a7862d6b7be004b660 in term 0.
I20260927 07:06:49.193557 3451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.188551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.30.16.67:38565)
0927 07:06:49.190215 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.190233 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:49.190386 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:49.190835 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:49.190838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:49.190854 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:49.191823 (+ 969us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:06:49.191827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:49.191842 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:06:49.191885 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:49.192295 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:49.192299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:49.192303 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 07:06:49.193034 (+ 731us) client_negotiation.cc:240] Negotiation successful
0927 07:06:49.193086 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"thread_start_us":1007,"threads_started":1}
I20260927 07:06:49.194146 3312 leader_election.cc:302] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3a4b71a5c244a7862d6b7be004b660, db80a9dcb7b0411799f6bcbb0022eea3; no voters:
I20260927 07:06:49.194646 3448 raft_consensus.cc:2781] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:06:49.194499 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979a48ab842485e86d4ef89234613ee" candidate_uuid: "5f3a4b71a5c244a7862d6b7be004b660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45874e1761fa4033aa6e89dfe80555df" is_pre_election: true
I20260927 07:06:49.194800 3448 raft_consensus.cc:482] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:06:49.194900 3448 raft_consensus.cc:3037] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:49.195129 3117 raft_consensus.cc:2454] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f3a4b71a5c244a7862d6b7be004b660 in term 0.
I20260927 07:06:49.197739 3448 raft_consensus.cc:504] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.198372 3448 leader_election.cc:288] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [CANDIDATE]: Term 1 election: Requested vote from peers db80a9dcb7b0411799f6bcbb0022eea3 (127.30.16.66:39013), 45874e1761fa4033aa6e89dfe80555df (127.30.16.65:38539)
I20260927 07:06:49.198836 3117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979a48ab842485e86d4ef89234613ee" candidate_uuid: "5f3a4b71a5c244a7862d6b7be004b660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45874e1761fa4033aa6e89dfe80555df"
I20260927 07:06:49.198948 3248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b979a48ab842485e86d4ef89234613ee" candidate_uuid: "5f3a4b71a5c244a7862d6b7be004b660" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db80a9dcb7b0411799f6bcbb0022eea3"
I20260927 07:06:49.199056 3117 raft_consensus.cc:3037] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:49.199150 3248 raft_consensus.cc:3037] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:49.202461 3117 raft_consensus.cc:2454] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3a4b71a5c244a7862d6b7be004b660 in term 1.
I20260927 07:06:49.202457 3248 raft_consensus.cc:2454] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f3a4b71a5c244a7862d6b7be004b660 in term 1.
I20260927 07:06:49.203018 3313 leader_election.cc:302] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45874e1761fa4033aa6e89dfe80555df, 5f3a4b71a5c244a7862d6b7be004b660; no voters:
I20260927 07:06:49.203459 3448 raft_consensus.cc:2781] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:49.203651 3448 raft_consensus.cc:686] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 1 LEADER]: Becoming Leader. State: Replica: 5f3a4b71a5c244a7862d6b7be004b660, State: Running, Role: LEADER
I20260927 07:06:49.204263 3448 consensus_queue.cc:237] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } }
I20260927 07:06:49.209748 2975 catalog_manager.cc:5495] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f3a4b71a5c244a7862d6b7be004b660 (127.30.16.67). New cstate: current_term: 1 leader_uuid: "5f3a4b71a5c244a7862d6b7be004b660" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f3a4b71a5c244a7862d6b7be004b660" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 } health_report { overall_health: UNKNOWN } } }
I20260927 07:06:49.225857 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.223382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.30.16.67:43917)
0927 07:06:49.223552 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.223557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.223810 (+ 253us) server_negotiation.cc:407] Connection header received
0927 07:06:49.224061 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.224065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.224145 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.224263 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.225217 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.225223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.225228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.225249 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.225282 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.225294 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.225422 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.225556 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.225617 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 07:06:49.234861 3117 raft_consensus.cc:1261] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 1 FOLLOWER]: Refusing update from remote peer 5f3a4b71a5c244a7862d6b7be004b660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:49.234905 3248 raft_consensus.cc:1261] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 1 FOLLOWER]: Refusing update from remote peer 5f3a4b71a5c244a7862d6b7be004b660: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:06:49.235927 3450 consensus_queue.cc:1035] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45874e1761fa4033aa6e89dfe80555df" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:49.236270 3448 consensus_queue.cc:1035] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db80a9dcb7b0411799f6bcbb0022eea3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:06:49.249607 3454 mvcc.cc:205] Tried to move back new op lower bound from 7333858546615046144 to 7333858546506346496. Current Snapshot: MvccSnapshot[applied={T|T < 7333858546615046144}]
I20260927 07:06:49.280973 2975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:49.281317 2975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:06:49.285748 2975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d84616321744316858eddbeaae6c029: Sending DeleteTablet for 3 replicas of tablet b979a48ab842485e86d4ef89234613ee
I20260927 07:06:49.287158 3228 tablet_service.cc:1514] Processing DeleteTablet for tablet b979a48ab842485e86d4ef89234613ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:49 UTC) from {username='slave'} at 127.0.0.1:56472
I20260927 07:06:49.287922 3358 tablet_service.cc:1514] Processing DeleteTablet for tablet b979a48ab842485e86d4ef89234613ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:49 UTC) from {username='slave'} at 127.0.0.1:55162
I20260927 07:06:49.288661 3097 tablet_service.cc:1514] Processing DeleteTablet for tablet b979a48ab842485e86d4ef89234613ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:06:49 UTC) from {username='slave'} at 127.0.0.1:49940
I20260927 07:06:49.292327 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3035
I20260927 07:06:49.296713 3466 tablet_replica.cc:330] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: stopping tablet replica
I20260927 07:06:49.299203 3465 tablet_replica.cc:330] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: stopping tablet replica
I20260927 07:06:49.299475 3466 raft_consensus.cc:2229] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:49.300434 3466 raft_consensus.cc:2258] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:49.300953 3438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.293811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.30.16.65:38539)
0927 07:06:49.293986 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.293992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.295992 (+ 2000us) server_negotiation.cc:407] Connection header received
0927 07:06:49.297539 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.297547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.297673 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.297776 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.299979 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.299988 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.299996 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.300021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.300070 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.300090 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.300232 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.300473 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.300545 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:06:49.301347 3464 tablet_replica.cc:330] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: stopping tablet replica
I20260927 07:06:49.301813 3464 raft_consensus.cc:2229] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:06:49.302291 3464 raft_consensus.cc:2258] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:49.303524 3465 raft_consensus.cc:2229] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:06:49.303599 3157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:49.304221 3465 raft_consensus.cc:2258] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:06:49.307376 3464 ts_tablet_manager.cc:1905] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:49.313060 3465 ts_tablet_manager.cc:1905] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:49.314962 3464 ts_tablet_manager.cc:1918] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:49.315189 3464 log.cc:1196] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/b979a48ab842485e86d4ef89234613ee
I20260927 07:06:49.315810 3464 ts_tablet_manager.cc:1939] T b979a48ab842485e86d4ef89234613ee P db80a9dcb7b0411799f6bcbb0022eea3: Deleting consensus metadata
I20260927 07:06:49.317878 2961 catalog_manager.cc:4841] TS db80a9dcb7b0411799f6bcbb0022eea3 (127.30.16.66:39013): tablet b979a48ab842485e86d4ef89234613ee (table test-table [id=8a88e0984f494c8ebf8e3b2b0e1480fa]) successfully deleted
I20260927 07:06:49.321447 3465 ts_tablet_manager.cc:1918] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:06:49.321640 3465 log.cc:1196] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/b979a48ab842485e86d4ef89234613ee
I20260927 07:06:49.322281 3465 ts_tablet_manager.cc:1939] T b979a48ab842485e86d4ef89234613ee P 5f3a4b71a5c244a7862d6b7be004b660: Deleting consensus metadata
I20260927 07:06:49.324574 2961 catalog_manager.cc:4841] TS 5f3a4b71a5c244a7862d6b7be004b660 (127.30.16.67:43917): tablet b979a48ab842485e86d4ef89234613ee (table test-table [id=8a88e0984f494c8ebf8e3b2b0e1480fa]) successfully deleted
I20260927 07:06:49.305337 3466 ts_tablet_manager.cc:1905] T b979a48ab842485e86d4ef89234613ee P 45874e1761fa4033aa6e89dfe80555df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:06:49.401136 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3035
W20260927 07:06:49.427142 2962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.16.65:38539 (error 108)
I20260927 07:06:49.427944 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3166
I20260927 07:06:49.429366 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.428878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60802)
0927 07:06:49.429113 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.429224 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260927 07:06:49.429629 2962 catalog_manager.cc:4584] TS 45874e1761fa4033aa6e89dfe80555df (127.30.16.65:38539): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b979a48ab842485e86d4ef89234613ee: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
I20260927 07:06:49.431375 3436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.428764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.30.16.66:39013)
0927 07:06:49.428960 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.428967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.429292 (+ 325us) server_negotiation.cc:407] Connection header received
0927 07:06:49.429371 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.429377 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.429488 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.429591 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.430816 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.430820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.430824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.430839 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.430869 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.430882 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.431007 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.431118 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.431164 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:06:49.432019 3288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:49.483604 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.482934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60806)
0927 07:06:49.483187 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.483319 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 07:06:49.484789 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.484387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60818)
0927 07:06:49.484555 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.484640 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 07:06:49.522039 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3166
I20260927 07:06:49.540357 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3296
I20260927 07:06:49.543681 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.541374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.30.16.67:43917)
0927 07:06:49.541567 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:49.541573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:49.541595 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:06:49.541738 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:49.541743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:49.541804 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:49.541919 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:49.542908 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:49.542913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:49.542918 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:49.542936 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:49.542981 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:49.542993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:49.543148 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:49.543236 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 07:06:49.543299 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:06:49.544646 3418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:49.545462 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.544805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60820)
0927 07:06:49.545074 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.545233 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 07:06:49.546952 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.546290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60836)
0927 07:06:49.546463 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.546800 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 07:06:49.627731 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.627163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60850)
0927 07:06:49.627443 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.627558 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 07:06:49.628970 3440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:49.628514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38539 (local address 127.0.0.1:60862)
0927 07:06:49.628732 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:49.628813 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 07:06:49.666709 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3296
I20260927 07:06:49.686506 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 2945
I20260927 07:06:49.687860 3006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:49.794171 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 2945
2026-09-27T07:06:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2691 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 07:06:49.835851 30785 mini_oidc.cc:152] Starting JWKS server
I20260927 07:06:49.838442 30785 webserver.cc:460] Webserver started at https://127.0.0.1:41131/ using document root <none> and password file <none>
I20260927 07:06:49.838840 30785 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:06:49.839502 30785 webserver.cc:460] Webserver started at http://127.0.0.1:44153/ using document root <none> and password file <none>
2026-09-27T07:06:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:49Z Disabled control of system clock
I20260927 07:06:49.867352 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37991
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:37991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/testchainca.cert with env {}
W20260927 07:06:50.195971 3478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:50.196270 3478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:50.196358 3478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:50.203814 3478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:50.203933 3478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:50.203975 3478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:50.204032 3478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:50.204077 3478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:06:50.204133 3478 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:06:50.204176 3478 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41131/jwks.json
I20260927 07:06:50.214356 3478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:37991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:50.215688 3478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:50.217003 3478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:50.228919 3486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:50.228996 3484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:50.229005 3483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:50.230206 3478 server_base.cc:1034] running on GCE node
I20260927 07:06:50.231038 3478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:50.232414 3478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:50.233609 3478 hybrid_clock.cc:648] HybridClock initialized: now 1790492810233550 us; error 57 us; skew 500 ppm
I20260927 07:06:50.233960 3478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:50.236191 3478 webserver.cc:460] Webserver started at http://127.30.16.126:42161/ using document root <none> and password file <none>
I20260927 07:06:50.236865 3478 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:50.236982 3478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:50.237293 3478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:50.240195 3478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "ca01b6daacbe41eb8e57c46c181ac928"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "2d31bc7907defbc36598b4d7b5c827db"
server_key_iv: "765e872afa69c6f347ab58c1fe10fb2e"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.240976 3478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "ca01b6daacbe41eb8e57c46c181ac928"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "2d31bc7907defbc36598b4d7b5c827db"
server_key_iv: "765e872afa69c6f347ab58c1fe10fb2e"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.246080 3478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:06:50.249651 3492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:50.251219 3478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:50.251415 3478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "ca01b6daacbe41eb8e57c46c181ac928"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "2d31bc7907defbc36598b4d7b5c827db"
server_key_iv: "765e872afa69c6f347ab58c1fe10fb2e"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.251663 3478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:50.265689 3478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:50.266697 3478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:50.267033 3478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:50.285920 3478 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:37991
I20260927 07:06:50.285941 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:37991 every 8 connection(s)
I20260927 07:06:50.287366 3478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:50.291270 3544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:50.296901 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3478
I20260927 07:06:50.297130 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:50.297900 30785 external_mini_cluster.cc:1428] Setting key 071b96532df4d1e94fb29efd9fe20df1
I20260927 07:06:50.302856 3544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928: Bootstrap starting.
I20260927 07:06:50.306465 3544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:50.307638 3544 log.cc:824] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:50.310745 3544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928: No bootstrap required, opened a new log
I20260927 07:06:50.312685 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:50.300223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.30.16.126:37991)
0927 07:06:50.301357 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:50.301369 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:50.301416 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:06:50.303362 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:50.303389 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:50.303898 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:50.304297 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:50.304978 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.306483 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.308113 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.308835 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.309043 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:50.310224 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:50.310232 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:50.310247 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:50.310891 (+ 644us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:50.310991 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:50.311012 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:50.311175 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:50.311393 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 07:06:50.311714 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":421,"threads_started":1}
I20260927 07:06:50.316869 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:06:50.319734 3544 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } }
I20260927 07:06:50.320242 3544 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:50.320354 3544 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca01b6daacbe41eb8e57c46c181ac928, State: Initialized, Role: FOLLOWER
I20260927 07:06:50.320930 3544 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } }
I20260927 07:06:50.321143 3544 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:50.321269 3544 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:50.321434 3544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:50.323895 3544 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } }
I20260927 07:06:50.324434 3544 leader_election.cc:302] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca01b6daacbe41eb8e57c46c181ac928; no voters:
I20260927 07:06:50.325001 3544 leader_election.cc:288] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:50.325167 3549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:50.325874 3549 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [term 1 LEADER]: Becoming Leader. State: Replica: ca01b6daacbe41eb8e57c46c181ac928, State: Running, Role: LEADER
I20260927 07:06:50.326481 3549 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } }
I20260927 07:06:50.327303 3544 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:50.329496 3550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca01b6daacbe41eb8e57c46c181ac928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } } }
I20260927 07:06:50.329567 3551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca01b6daacbe41eb8e57c46c181ac928. Latest consensus state: current_term: 1 leader_uuid: "ca01b6daacbe41eb8e57c46c181ac928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca01b6daacbe41eb8e57c46c181ac928" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37991 } } }
I20260927 07:06:50.329900 3550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:50.329960 3551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:50.333065 3556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:50.337534 3556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:50.346689 3556 catalog_manager.cc:1347] Generated new cluster ID: 96322c49b86a44e4b0bb9fd45bf8627b
I20260927 07:06:50.346827 3556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:50.363512 3556 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:50.365379 3556 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:50.376379 3556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca01b6daacbe41eb8e57c46c181ac928: Generated new TSK 0
I20260927 07:06:50.377295 3556 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 07:06:50.421481 3574 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 07:06:50.421705 3574 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 07:06:50.422914 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:50.400409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.30.16.126:37991)
0927 07:06:50.400615 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:50.400621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:50.401067 (+ 446us) server_negotiation.cc:407] Connection header received
0927 07:06:50.401363 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:50.401369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:50.401466 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:50.401627 (+ 161us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:06:50.402164 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.403357 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.404894 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.405618 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.405742 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:50.406605 (+ 863us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:06:50.422100 (+ 15495us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 07:06:50.422597 (+ 497us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 07:06:50.423059 3547 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57040: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 07:06:50.424945 3573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:50.400283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37991 (local address 127.0.0.1:57040)
0927 07:06:50.400961 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:50.400983 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:50.401133 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:50.401655 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:50.401659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:50.401671 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:06:50.402021 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:50.402030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.403547 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.403552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:50.404724 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:50.404738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:50.405791 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:50.405795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:50.406183 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:50.406270 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:06:50.422592 (+ 16322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 07:06:50.423384 (+ 792us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.16.126:37991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":356,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.30.16.126:37991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41131/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 07:06:50.426633 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3478
I20260927 07:06:50.427969 3539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:50.522864 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3478
2026-09-27T07:06:50Z chronyd exiting
I20260927 07:06:50.575289 30785 test_util.cc:175] -----------------------------------------------
I20260927 07:06:50.575461 30785 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790492767955610-30785-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 07:06:50.579291 30785 mini_oidc.cc:152] Starting JWKS server
I20260927 07:06:50.581411 30785 webserver.cc:460] Webserver started at https://127.0.0.1:36461/ using document root <none> and password file <none>
I20260927 07:06:50.581665 30785 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:06:50.582401 30785 webserver.cc:460] Webserver started at http://127.0.0.1:39839/ using document root <none> and password file <none>
2026-09-27T07:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:50Z Disabled control of system clock
I20260927 07:06:50.611815 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34109
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:40487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:34109
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/testchainca.cert with env {}
W20260927 07:06:50.939703 3581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:50.940027 3581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:50.940117 3581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:50.947492 3581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:50.947626 3581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:50.947664 3581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:50.947715 3581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:50.947765 3581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:06:50.947825 3581 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:06:50.947865 3581 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36461/jwks.json
I20260927 07:06:50.958160 3581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:34109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:50.959478 3581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:50.960842 3581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:50.972971 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:50.972926 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:50.973206 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:50.973875 3581 server_base.cc:1034] running on GCE node
I20260927 07:06:50.974700 3581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:50.976143 3581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:50.977373 3581 hybrid_clock.cc:648] HybridClock initialized: now 1790492810977334 us; error 37 us; skew 500 ppm
I20260927 07:06:50.977715 3581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:50.980311 3581 webserver.cc:460] Webserver started at http://127.30.16.126:36685/ using document root <none> and password file <none>
I20260927 07:06:50.981043 3581 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:50.981135 3581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:50.981411 3581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:50.984752 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "1dbadb95b8e56d09865172298e5a9692"
server_key_iv: "be1363987aac1a9f80bd2b031301dd02"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.985550 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "1dbadb95b8e56d09865172298e5a9692"
server_key_iv: "be1363987aac1a9f80bd2b031301dd02"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.991093 3581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:06:50.994916 3595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:50.996542 3581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:50.996745 3581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29"
format_stamp: "Formatted at 2026-09-27 07:06:50 on dist-test-slave-8vqf"
server_key: "1dbadb95b8e56d09865172298e5a9692"
server_key_iv: "be1363987aac1a9f80bd2b031301dd02"
server_key_version: "encryptionkey@0"
I20260927 07:06:50.996990 3581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:51.029650 3581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:51.030616 3581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:51.030936 3581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:51.049500 3581 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:34109
I20260927 07:06:51.049525 3646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:34109 every 8 connection(s)
I20260927 07:06:51.050938 3581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:51.051822 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3581
I20260927 07:06:51.052014 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:51.052780 30785 external_mini_cluster.cc:1428] Setting key 3790f1bf92cf4723ac7b5803a470bcb8
I20260927 07:06:51.055518 3647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:51.068938 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:51.055017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.30.16.126:34109)
0927 07:06:51.056222 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:51.056235 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:51.056287 (+ 52us) server_negotiation.cc:407] Connection header received
0927 07:06:51.058461 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:51.058489 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:51.059154 (+ 665us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:51.059615 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:51.060186 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.062265 (+ 2079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.063706 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.064710 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.064903 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.066012 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:51.066023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:51.066041 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:51.066835 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:51.066942 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:51.066968 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:51.067093 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:51.067295 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 07:06:51.067664 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":356,"threads_started":1}
I20260927 07:06:51.069852 3647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29: Bootstrap starting.
I20260927 07:06:51.073098 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:06:51.073746 3647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:51.075001 3647 log.cc:824] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:51.078145 3647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29: No bootstrap required, opened a new log
I20260927 07:06:51.085794 3647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } }
I20260927 07:06:51.086149 3647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:51.086223 3647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a4b5cc01e07421cbadc6dc48c0b0d29, State: Initialized, Role: FOLLOWER
I20260927 07:06:51.086767 3647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } }
I20260927 07:06:51.086969 3647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:51.087062 3647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:51.087191 3647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:51.089262 3647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } }
I20260927 07:06:51.089736 3647 leader_election.cc:302] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a4b5cc01e07421cbadc6dc48c0b0d29; no voters:
I20260927 07:06:51.090381 3647 leader_election.cc:288] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:51.090533 3652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:51.091358 3652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [term 1 LEADER]: Becoming Leader. State: Replica: 0a4b5cc01e07421cbadc6dc48c0b0d29, State: Running, Role: LEADER
I20260927 07:06:51.091867 3652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } }
I20260927 07:06:51.092656 3647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:51.094894 3653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } } }
I20260927 07:06:51.095347 3653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:51.096474 3654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a4b5cc01e07421cbadc6dc48c0b0d29. Latest consensus state: current_term: 1 leader_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a4b5cc01e07421cbadc6dc48c0b0d29" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34109 } } }
I20260927 07:06:51.096912 3654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29 [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:51.097349 3661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:51.102748 3661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:51.116739 3661 catalog_manager.cc:1347] Generated new cluster ID: c9553d67cfb74514bd7dcab2d73bfc51
I20260927 07:06:51.116871 3661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:51.149405 3661 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:51.150832 3661 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:51.158226 3661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a4b5cc01e07421cbadc6dc48c0b0d29: Generated new TSK 0
I20260927 07:06:51.159399 3661 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 07:06:51.204746 3677 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 07:06:51.204891 3677 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 07:06:51.206182 3650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:51.185139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.30.16.126:34109)
0927 07:06:51.185345 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:51.185352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:51.185807 (+ 455us) server_negotiation.cc:407] Connection header received
0927 07:06:51.186043 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:51.186049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:51.186143 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:51.186319 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:06:51.186853 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.187994 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.189406 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.190121 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.190266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.190848 (+ 582us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:06:51.205461 (+ 14613us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 07:06:51.205845 (+ 384us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 07:06:51.206305 3676 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:51.184937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34109 (local address 127.0.0.1:40852)
0927 07:06:51.185695 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:51.185711 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:51.185839 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:51.186335 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:51.186338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:51.186348 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:06:51.186677 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:51.186687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.188137 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.188140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:51.189263 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:51.189274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.190311 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.190315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:51.190455 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.190510 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:06:51.205723 (+ 15213us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 07:06:51.205982 (+ 259us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.16.126:34109: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":447,"threads_started":1}
W20260927 07:06:51.206317 3650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 07:06:51.207681 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3581
I20260927 07:06:51.208832 3642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:51.306407 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3581
2026-09-27T07:06:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (783 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 07:06:51.362349 30785 mini_oidc.cc:152] Starting JWKS server
I20260927 07:06:51.364245 30785 webserver.cc:460] Webserver started at https://127.0.0.1:44793/ using document root <none> and password file <none>
I20260927 07:06:51.364465 30785 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 07:06:51.365037 30785 webserver.cc:460] Webserver started at http://127.0.0.1:42005/ using document root <none> and password file <none>
2026-09-27T07:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:51Z Disabled control of system clock
I20260927 07:06:51.392866 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44107
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:44107
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44793/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 07:06:51.719805 3684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:51.720127 3684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:51.720216 3684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:51.727392 3684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:51.727504 3684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:51.727541 3684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:51.727589 3684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:51.727638 3684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:06:51.727698 3684 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:06:51.727737 3684 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44793/jwks.json
I20260927 07:06:51.736922 3684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:44107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44793/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:51.738193 3684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:51.740108 3684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:51.752123 3689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:51.752132 3692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:51.752105 3690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:51.752794 3684 server_base.cc:1034] running on GCE node
I20260927 07:06:51.753608 3684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:51.755136 3684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:51.756413 3684 hybrid_clock.cc:648] HybridClock initialized: now 1790492811756334 us; error 53 us; skew 500 ppm
I20260927 07:06:51.756798 3684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:51.759312 3684 webserver.cc:460] Webserver started at http://127.30.16.126:40733/ using document root <none> and password file <none>
I20260927 07:06:51.760062 3684 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:51.760196 3684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:51.760532 3684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:51.763650 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "4e69c9efebd246e797c185712c0cba9e"
format_stamp: "Formatted at 2026-09-27 07:06:51 on dist-test-slave-8vqf"
server_key: "02ae38cd2d5983825efb8df05d9e264f"
server_key_iv: "70b27670f720eafc7d41a5d0e4b4347e"
server_key_version: "encryptionkey@0"
I20260927 07:06:51.764519 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "4e69c9efebd246e797c185712c0cba9e"
format_stamp: "Formatted at 2026-09-27 07:06:51 on dist-test-slave-8vqf"
server_key: "02ae38cd2d5983825efb8df05d9e264f"
server_key_iv: "70b27670f720eafc7d41a5d0e4b4347e"
server_key_version: "encryptionkey@0"
I20260927 07:06:51.769667 3684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 07:06:51.773538 3698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:51.775141 3684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:51.775341 3684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "4e69c9efebd246e797c185712c0cba9e"
format_stamp: "Formatted at 2026-09-27 07:06:51 on dist-test-slave-8vqf"
server_key: "02ae38cd2d5983825efb8df05d9e264f"
server_key_iv: "70b27670f720eafc7d41a5d0e4b4347e"
server_key_version: "encryptionkey@0"
I20260927 07:06:51.775563 3684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:51.802903 3684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:51.803793 3684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:51.804214 3684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:51.822595 3684 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:44107
I20260927 07:06:51.822635 3749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:44107 every 8 connection(s)
I20260927 07:06:51.824003 3684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:51.827984 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:51.832854 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3684
I20260927 07:06:51.833078 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:51.833703 30785 external_mini_cluster.cc:1428] Setting key 288412e70773a9a874d1a7da77b40c65
I20260927 07:06:51.839993 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e: Bootstrap starting.
I20260927 07:06:51.843874 3750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:51.845028 3750 log.cc:824] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:51.848165 3750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e: No bootstrap required, opened a new log
I20260927 07:06:51.849330 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:51.835752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.30.16.126:44107)
0927 07:06:51.836959 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:51.836974 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:51.837044 (+ 70us) server_negotiation.cc:407] Connection header received
0927 07:06:51.839348 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:51.839376 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:51.839993 (+ 617us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:51.840461 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:51.840991 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.843048 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.844596 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.845719 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.845910 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.847105 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:51.847113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:51.847128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:51.847657 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:51.847731 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:51.847750 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:51.847886 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:51.848100 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 07:06:51.848429 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":376,"threads_started":1}
I20260927 07:06:51.853101 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:06:51.856966 3750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } }
I20260927 07:06:51.857376 3750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:51.857493 3750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e69c9efebd246e797c185712c0cba9e, State: Initialized, Role: FOLLOWER
I20260927 07:06:51.858121 3750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } }
I20260927 07:06:51.858384 3750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:51.858541 3750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:51.858809 3750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:51.861456 3750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } }
I20260927 07:06:51.861985 3750 leader_election.cc:302] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e69c9efebd246e797c185712c0cba9e; no voters:
I20260927 07:06:51.862610 3750 leader_election.cc:288] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:51.862771 3755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:51.863557 3755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [term 1 LEADER]: Becoming Leader. State: Replica: 4e69c9efebd246e797c185712c0cba9e, State: Running, Role: LEADER
I20260927 07:06:51.864212 3755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } }
I20260927 07:06:51.865077 3750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:51.867455 3757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e69c9efebd246e797c185712c0cba9e. Latest consensus state: current_term: 1 leader_uuid: "4e69c9efebd246e797c185712c0cba9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } } }
I20260927 07:06:51.867858 3757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:51.868515 3756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e69c9efebd246e797c185712c0cba9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e69c9efebd246e797c185712c0cba9e" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44107 } } }
I20260927 07:06:51.869023 3756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:51.869982 3764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:51.874461 3764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:51.883410 3764 catalog_manager.cc:1347] Generated new cluster ID: bbd3d4c6f6e7434fbc8331789f63d059
I20260927 07:06:51.883550 3764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:51.916755 3764 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:51.918615 3764 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:51.929284 3764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e69c9efebd246e797c185712c0cba9e: Generated new TSK 0
I20260927 07:06:51.930222 3764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:06:51.959693 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:51.951387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.30.16.126:44107)
0927 07:06:51.951598 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:51.951605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:51.952021 (+ 416us) server_negotiation.cc:407] Connection header received
0927 07:06:51.952285 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:51.952290 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:51.952371 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:51.952535 (+ 164us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:06:51.952997 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.953750 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.955204 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.955641 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.955791 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.956281 (+ 490us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:06:51.958998 (+ 2717us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44793/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 07:06:51.959327 (+ 329us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44793/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 07:06:51.959832 3753 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33506: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44793/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 07:06:51.961887 3779 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:51.951246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:44107 (local address 127.0.0.1:33506)
0927 07:06:51.951895 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:51.951911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:51.952093 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:51.952551 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:51.952556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:51.952567 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:06:51.952833 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:51.952842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.953900 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.953912 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:51.955049 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:51.955059 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:51.955837 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:51.955845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:51.956001 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:51.956035 (+ 34us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:06:51.959222 (+ 3187us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44793/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 07:06:51.961427 (+ 2205us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.16.126:44107: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44793/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":1863,"thread_start_us":357,"threads_started":1}
I20260927 07:06:51.962913 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3684
I20260927 07:06:51.963990 3745 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 07:06:51.971769 3780 webserver.cc:533] Webserver: sslize failed:
E20260927 07:06:51.971894 3780 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 07:06:52.059365 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3684
2026-09-27T07:06:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (754 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T07:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:52Z Disabled control of system clock
I20260927 07:06:52.145058 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34893
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:40339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:34893
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/testchainca.cert with env {}
W20260927 07:06:52.482779 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:52.483181 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:52.483325 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:52.491298 3785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:52.491411 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:52.491472 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:52.491554 3785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:52.491632 3785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 07:06:52.491720 3785 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 07:06:52.491792 3785 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 07:06:52.501926 3785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:34893
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:52.503330 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:52.504649 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:52.517186 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:52.517274 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:52.517705 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:52.518085 3785 server_base.cc:1034] running on GCE node
I20260927 07:06:52.519019 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:52.520454 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:52.521709 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790492812521664 us; error 38 us; skew 500 ppm
I20260927 07:06:52.522074 3785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:06:52.524574 3785 webserver.cc:460] Webserver started at http://127.30.16.126:39211/ using document root <none> and password file <none>
I20260927 07:06:52.525302 3785 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:52.525450 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:52.525761 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:52.528753 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "4acd28cd32c04f33851b9a69c0fd95dc"
format_stamp: "Formatted at 2026-09-27 07:06:52 on dist-test-slave-8vqf"
server_key: "adddaa2d4c0e93b0b0c9615bf6d0f3d6"
server_key_iv: "929cedd0f9c6c57335e8c504b7c52f46"
server_key_version: "encryptionkey@0"
I20260927 07:06:52.529569 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "4acd28cd32c04f33851b9a69c0fd95dc"
format_stamp: "Formatted at 2026-09-27 07:06:52 on dist-test-slave-8vqf"
server_key: "adddaa2d4c0e93b0b0c9615bf6d0f3d6"
server_key_iv: "929cedd0f9c6c57335e8c504b7c52f46"
server_key_version: "encryptionkey@0"
I20260927 07:06:52.535326 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:06:52.540339 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:52.541879 3785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260927 07:06:52.542064 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "4acd28cd32c04f33851b9a69c0fd95dc"
format_stamp: "Formatted at 2026-09-27 07:06:52 on dist-test-slave-8vqf"
server_key: "adddaa2d4c0e93b0b0c9615bf6d0f3d6"
server_key_iv: "929cedd0f9c6c57335e8c504b7c52f46"
server_key_version: "encryptionkey@0"
I20260927 07:06:52.542291 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:52.556071 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:52.556910 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:52.557190 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:52.576061 3785 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:34893
I20260927 07:06:52.576079 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:34893 every 8 connection(s)
I20260927 07:06:52.577502 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:52.581491 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:52.587213 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3785
I20260927 07:06:52.587430 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:52.588232 30785 external_mini_cluster.cc:1428] Setting key 87f780076624b99a9ae34b71dcfad9fc
I20260927 07:06:52.593287 3851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc: Bootstrap starting.
I20260927 07:06:52.596990 3851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:52.598258 3851 log.cc:824] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:52.601395 3851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc: No bootstrap required, opened a new log
I20260927 07:06:52.609691 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:52.590377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.30.16.126:34893)
0927 07:06:52.591600 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:52.591612 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:52.591654 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:06:52.593294 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:52.593316 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:52.593867 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:52.594204 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:52.594867 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.596164 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.598042 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.598977 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.599187 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:52.606538 (+ 7351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:52.606570 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:52.606597 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:06:52.607363 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:52.607476 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:06:52.607509 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:52.607641 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:52.607848 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 07:06:52.608307 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":499,"threads_started":1}
I20260927 07:06:52.609862 3851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } }
I20260927 07:06:52.610281 3851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:52.610407 3851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4acd28cd32c04f33851b9a69c0fd95dc, State: Initialized, Role: FOLLOWER
I20260927 07:06:52.610998 3851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } }
I20260927 07:06:52.611245 3851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:52.611409 3851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:52.611629 3851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:52.613883 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:06:52.614321 3851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } }
I20260927 07:06:52.614926 3851 leader_election.cc:302] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4acd28cd32c04f33851b9a69c0fd95dc; no voters:
I20260927 07:06:52.615536 3851 leader_election.cc:288] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:52.615679 3856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:52.616875 3856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [term 1 LEADER]: Becoming Leader. State: Replica: 4acd28cd32c04f33851b9a69c0fd95dc, State: Running, Role: LEADER
I20260927 07:06:52.617527 3856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } }
I20260927 07:06:52.617832 3851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:52.621374 3858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4acd28cd32c04f33851b9a69c0fd95dc. Latest consensus state: current_term: 1 leader_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } } }
I20260927 07:06:52.621626 3857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4acd28cd32c04f33851b9a69c0fd95dc" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34893 } } }
I20260927 07:06:52.622030 3857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:52.622035 3858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:52.631359 3864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:52.637696 3864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:52.646696 3864 catalog_manager.cc:1347] Generated new cluster ID: d7b10a407cdf4a75a7e188a71af46fea
I20260927 07:06:52.646828 3864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:52.676028 3864 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:52.677317 3864 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:52.682629 3864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4acd28cd32c04f33851b9a69c0fd95dc: Generated new TSK 0
I20260927 07:06:52.683698 3864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:06:52.871840 3854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:52.708463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38014 (local address 127.30.16.126:34893)
0927 07:06:52.708682 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:52.708688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:52.709007 (+ 319us) server_negotiation.cc:407] Connection header received
0927 07:06:52.709250 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:52.709254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:52.709327 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:52.709475 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0927 07:06:52.709974 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.710831 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.712382 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.712811 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.712947 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:52.713407 (+ 460us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 07:06:52.870987 (+157580us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 07:06:52.871467 (+ 480us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":89}
W20260927 07:06:52.872000 3854 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 07:06:52.872123 3880 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:52.708254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34893 (local address 127.0.0.1:38014)
0927 07:06:52.708894 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:52.708915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:52.709068 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:52.709522 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:52.709527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:52.709540 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0927 07:06:52.709826 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:52.709838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.711035 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.711039 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:52.712228 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:52.712239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:52.712990 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:52.712996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:52.713139 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:52.713172 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 07:06:52.871300 (+158128us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 07:06:52.871612 (+ 312us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.30.16.126:34893: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":312,"threads_started":1}
I20260927 07:06:52.873364 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3785
I20260927 07:06:52.874830 3846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:52.981719 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3785
2026-09-27T07:06:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (900 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (414 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (419 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:53 dist-test-slave-8vqf krb5kdc[3895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:53 dist-test-slave-8vqf krb5kdc[3895](info): set up 2 sockets
Sep 27 07:06:53 dist-test-slave-8vqf krb5kdc[3895](info): commencing operation
krb5kdc: starting...
W20260927 07:06:54.931348 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.010s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:55 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:06:55.146740 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:45209
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:55.480573 3908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:55.480959 3908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:55.481117 3908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:55.488550 3908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:55.488713 3908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:55.488826 3908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:55.488933 3908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:55.489012 3908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:55.499085 3908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:55.500468 3908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:55.501814 3908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:55.514057 3913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.514196 3914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:55.514135 3916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:55.515220 3908 server_base.cc:1034] running on GCE node
I20260927 07:06:55.516121 3908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:55.517742 3908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:55.518989 3908 hybrid_clock.cc:648] HybridClock initialized: now 1790492815518924 us; error 48 us; skew 500 ppm
I20260927 07:06:55.519441 3908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:55 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:55.523099 3908 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:06:55.524621 3908 webserver.cc:460] Webserver started at http://127.30.16.126:44197/ using document root <none> and password file <none>
I20260927 07:06:55.525342 3908 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:55.525497 3908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:55.525823 3908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:55.528913 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "1dc56d79bca941039b92e21b3be67e3c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-8vqf"
server_key: "1f6a56c2cd6d4d8e40fdb58c7f5e9581"
server_key_iv: "dbc4f06ec4a65fdc272877ab0e9a6247"
server_key_version: "encryptionkey@0"
I20260927 07:06:55.529704 3908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "1dc56d79bca941039b92e21b3be67e3c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-8vqf"
server_key: "1f6a56c2cd6d4d8e40fdb58c7f5e9581"
server_key_iv: "dbc4f06ec4a65fdc272877ab0e9a6247"
server_key_version: "encryptionkey@0"
I20260927 07:06:55.535213 3908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 07:06:55.538725 3923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:55.540215 3908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:06:55.540405 3908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "1dc56d79bca941039b92e21b3be67e3c"
format_stamp: "Formatted at 2026-09-27 07:06:55 on dist-test-slave-8vqf"
server_key: "1f6a56c2cd6d4d8e40fdb58c7f5e9581"
server_key_iv: "dbc4f06ec4a65fdc272877ab0e9a6247"
server_key_version: "encryptionkey@0"
I20260927 07:06:55.540628 3908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:55.567188 3908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:55.571564 3908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:55.571879 3908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:55.590852 3908 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:45209
I20260927 07:06:55.590874 3974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:45209 every 8 connection(s)
I20260927 07:06:55.592319 3908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:55.596271 3975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:55.597602 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 3908
I20260927 07:06:55.597829 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:55.598456 30785 external_mini_cluster.cc:1428] Setting key 35407ce8e74767a46ad79fa65574bfab
I20260927 07:06:55.608554 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c: Bootstrap starting.
I20260927 07:06:55.612840 3975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c: Neither blocks nor log segments found. Creating new log.
Sep 27 07:06:55 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:55.614039 3975 log.cc:824] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:55.617815 3975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c: No bootstrap required, opened a new log
I20260927 07:06:55.624176 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:55.600709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.30.16.126:45209)
0927 07:06:55.601533 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:55.601546 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:55.601588 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:06:55.603364 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:55.603387 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:55.603776 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:55.604106 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:55.605878 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.607393 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.609120 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:55.609827 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:55.609968 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:55.615266 (+ 5298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:55.615276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:55.615293 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:55.615332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:55.619253 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.620129 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.620139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.620148 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.620373 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:55.620729 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:55.620735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:55.620739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:55.621485 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:55.621775 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:55.622122 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 07:06:55.622680 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":357,"threads_started":1}
I20260927 07:06:55.627671 3975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } }
I20260927 07:06:55.628014 3975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:55.628101 3975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dc56d79bca941039b92e21b3be67e3c, State: Initialized, Role: FOLLOWER
I20260927 07:06:55.628650 3975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } }
I20260927 07:06:55.628849 3975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:55.628942 3975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:55.629076 3975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:55.631457 3975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } }
I20260927 07:06:55.632061 3975 leader_election.cc:302] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dc56d79bca941039b92e21b3be67e3c; no voters:
I20260927 07:06:55.632575 3975 leader_election.cc:288] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:55.632733 3980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:55.633425 3980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [term 1 LEADER]: Becoming Leader. State: Replica: 1dc56d79bca941039b92e21b3be67e3c, State: Running, Role: LEADER
I20260927 07:06:55.633919 3980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } }
I20260927 07:06:55.634696 3975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:55.637326 3981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dc56d79bca941039b92e21b3be67e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } } }
I20260927 07:06:55.637696 3981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:55.637413 3982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dc56d79bca941039b92e21b3be67e3c. Latest consensus state: current_term: 1 leader_uuid: "1dc56d79bca941039b92e21b3be67e3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dc56d79bca941039b92e21b3be67e3c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45209 } } }
I20260927 07:06:55.637822 3982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:55.640189 3985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:55.646469 3985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:55.655478 3985 catalog_manager.cc:1347] Generated new cluster ID: 114a5f62800b49dd8c8bbb76d94cad58
I20260927 07:06:55.655610 3985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:55.680027 3985 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:55.681384 3985 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:55.689566 3985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dc56d79bca941039b92e21b3be67e3c: Generated new TSK 0
I20260927 07:06:55.690442 3985 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:06:55.751155 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:56.087972 4001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:56.088323 4001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:56.088500 4001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:56.096037 4001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:56.096215 4001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:56.096403 4001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:06:56.106510 4001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:56.107890 4001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:56.109423 4001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:56.122987 4007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:56.123055 4006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:56.124178 4009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:56.124423 4001 server_base.cc:1034] running on GCE node
I20260927 07:06:56.125365 4001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:56.126356 4001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:56.127575 4001 hybrid_clock.cc:648] HybridClock initialized: now 1790492816127513 us; error 64 us; skew 500 ppm
I20260927 07:06:56.127923 4001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:56 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:56.131208 4001 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:06:56.132668 4001 webserver.cc:460] Webserver started at http://127.30.16.65:35673/ using document root <none> and password file <none>
I20260927 07:06:56.133360 4001 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:56.133459 4001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:56.133723 4001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:56.136833 4001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "5444ba2c19c64b358edca277e8b9aaae"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "4a95f340408534d26f1d241a85d3ed00"
server_key_iv: "2ca296b1b69293e4bb16e081cc185f8d"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.137555 4001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "5444ba2c19c64b358edca277e8b9aaae"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "4a95f340408534d26f1d241a85d3ed00"
server_key_iv: "2ca296b1b69293e4bb16e081cc185f8d"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.143194 4001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 07:06:56.147151 4016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.148995 4001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:56.149206 4001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "5444ba2c19c64b358edca277e8b9aaae"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "4a95f340408534d26f1d241a85d3ed00"
server_key_iv: "2ca296b1b69293e4bb16e081cc185f8d"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.149447 4001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:56.168648 4001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:56.173311 4001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:56.173661 4001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:56.174767 4001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:56.176216 4001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:56.176312 4001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.176502 4001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:56.176575 4001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.210862 4001 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45769
I20260927 07:06:56.210908 4128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45769 every 8 connection(s)
I20260927 07:06:56.212220 4001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:06:56.213975 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4001
I20260927 07:06:56.214162 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:06:56.214948 30785 external_mini_cluster.cc:1428] Setting key 60bfd96a6aaf1ef845370e30aff9c72a
Sep 27 07:06:56 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:06:56.235940 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.214988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60973 (local address 127.30.16.126:45209)
0927 07:06:56.215473 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:56.215480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:56.216741 (+ 1261us) server_negotiation.cc:407] Connection header received
0927 07:06:56.219543 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:56.219548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:56.219612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:56.219706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:56.221880 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.222725 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.225113 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.225518 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.225654 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.229435 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:56.229441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:56.229444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:56.229478 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:56.232654 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.233469 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.233477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.233481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.233606 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.234052 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.234058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.234061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.234334 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:56.234491 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:56.235318 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 07:06:56.235556 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":233,"threads_started":1}
I20260927 07:06:56.237082 4132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.215628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:45209 (local address 127.30.16.65:60973)
0927 07:06:56.216562 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:56.216619 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:56.219235 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:56.220001 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:56.220015 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:56.220834 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:56.221683 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.221711 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.222892 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.222900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.224954 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.224966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.225769 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.225776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.226023 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.226759 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:56.226786 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:56.229187 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:56.232830 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.232840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.232860 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.233327 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.233723 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.233728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.233732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.233935 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.234472 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:56.234481 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:56.234924 (+ 443us) client_negotiation.cc:769] Sending connection context
0927 07:06:56.235214 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 07:06:56.235850 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:06:56.238781 4129 heartbeater.cc:347] Connected to a master server at 127.30.16.126:45209
I20260927 07:06:56.239149 4129 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:56.239945 4129 heartbeater.cc:511] Master 127.30.16.126:45209 requested a full tablet report, sending...
I20260927 07:06:56.242272 3940 ts_manager.cc:194] Registered new tserver with Master: 5444ba2c19c64b358edca277e8b9aaae (127.30.16.65:45769)
I20260927 07:06:56.243968 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:60973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:06:56.266232 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:56.602722 4135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:56.603005 4135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:56.603163 4135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:56.610683 4135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:56.610795 4135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:56.611052 4135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:06:56.621047 4135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:56.622591 4135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:56.623935 4135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:56.637413 4140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:56.637492 4141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:56.637729 4143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:56.638913 4135 server_base.cc:1034] running on GCE node
I20260927 07:06:56.639526 4135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:56.640508 4135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:56.641726 4135 hybrid_clock.cc:648] HybridClock initialized: now 1790492816641663 us; error 60 us; skew 500 ppm
I20260927 07:06:56.642062 4135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:56 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:56.645223 4135 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:06:56.646958 4135 webserver.cc:460] Webserver started at http://127.30.16.66:46403/ using document root <none> and password file <none>
I20260927 07:06:56.647660 4135 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:56.647804 4135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:56.648118 4135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:56.651203 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "514dd608d7e84c4a98b22714ececea75"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "3e4685be63f7b89656a71ba80ccd2f12"
server_key_iv: "11b48ab1c145985db00e7248e4ae3ea4"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.652000 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "514dd608d7e84c4a98b22714ececea75"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "3e4685be63f7b89656a71ba80ccd2f12"
server_key_iv: "11b48ab1c145985db00e7248e4ae3ea4"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.657392 4135 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:06:56.660951 4150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.662472 4135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:06:56.662714 4135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "514dd608d7e84c4a98b22714ececea75"
format_stamp: "Formatted at 2026-09-27 07:06:56 on dist-test-slave-8vqf"
server_key: "3e4685be63f7b89656a71ba80ccd2f12"
server_key_iv: "11b48ab1c145985db00e7248e4ae3ea4"
server_key_version: "encryptionkey@0"
I20260927 07:06:56.662933 4135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:56.681396 4135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:56.685544 4135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:56.685853 4135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:56.686908 4135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:56.688331 4135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:56.688434 4135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.688594 4135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:56.688668 4135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:56.723129 4135 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:34207
I20260927 07:06:56.723173 4262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:34207 every 8 connection(s)
I20260927 07:06:56.724462 4135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:06:56.727171 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4135
I20260927 07:06:56.727365 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:06:56.727806 30785 external_mini_cluster.cc:1428] Setting key 146caf9449dd92bc7c8d318226e70538
Sep 27 07:06:56 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492816, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:56.746577 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.727271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:33549 (local address 127.30.16.126:45209)
0927 07:06:56.727504 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:56.727512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:56.728911 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 07:06:56.731305 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:56.731309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:56.731372 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:56.731454 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:56.733658 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.734337 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.736070 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.736503 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.736638 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.740413 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:56.740422 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:56.740428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:56.740476 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:56.743224 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.744077 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.744091 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.744093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.744206 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:56.744692 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:56.744696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:56.744700 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:56.745043 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:56.745189 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:56.745978 (+ 789us) server_negotiation.cc:299] Negotiation successful
0927 07:06:56.746204 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 07:06:56.747620 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:56.727838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:45209 (local address 127.30.16.66:33549)
0927 07:06:56.728716 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:56.728772 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:56.731005 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:56.731750 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:56.731766 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:56.732593 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:56.733484 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.733513 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.734580 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.734586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.735924 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:56.735935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:56.736731 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:56.736738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:56.736970 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:56.737684 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:56.737711 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:56.740128 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:56.743366 (+ 3238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.743375 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.743393 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.743957 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.744316 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:56.744321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:56.744325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:56.744580 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:56.745164 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:56.745172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:56.745580 (+ 408us) client_negotiation.cc:769] Sending connection context
0927 07:06:56.745871 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 07:06:56.746469 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":313,"threads_started":1}
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:06:56.749118 4263 heartbeater.cc:347] Connected to a master server at 127.30.16.126:45209
I20260927 07:06:56.749439 4263 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:06:56.750311 4263 heartbeater.cc:511] Master 127.30.16.126:45209 requested a full tablet report, sending...
I20260927 07:06:56.752007 3940 ts_manager.cc:194] Registered new tserver with Master: 514dd608d7e84c4a98b22714ececea75 (127.30.16.66:34207)
I20260927 07:06:56.752928 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:33549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:06:56.778854 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:57.109200 4268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:57.109496 4268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:57.109670 4268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:57.117254 4268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:57.117368 4268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:57.117566 4268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:06:57.127725 4268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:45209
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:57.129063 4268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:57.130404 4268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:57.143936 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:57.144062 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:57.143963 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:57.145224 4268 server_base.cc:1034] running on GCE node
I20260927 07:06:57.145869 4268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:57.146958 4268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:57.148192 4268 hybrid_clock.cc:648] HybridClock initialized: now 1790492817148133 us; error 46 us; skew 500 ppm
I20260927 07:06:57.148535 4268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.151921 4268 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:06:57.153501 4268 webserver.cc:460] Webserver started at http://127.30.16.67:39975/ using document root <none> and password file <none>
I20260927 07:06:57.154279 4268 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:57.154429 4268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:57.154778 4268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:57.157825 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "f68b78176da14edb9f10a1785ad804af"
format_stamp: "Formatted at 2026-09-27 07:06:57 on dist-test-slave-8vqf"
server_key: "60e2e0705ef90a27d0e354019fbd721a"
server_key_iv: "a20a7a5dccef46150d6c5ce08cf2debc"
server_key_version: "encryptionkey@0"
I20260927 07:06:57.158743 4268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "f68b78176da14edb9f10a1785ad804af"
format_stamp: "Formatted at 2026-09-27 07:06:57 on dist-test-slave-8vqf"
server_key: "60e2e0705ef90a27d0e354019fbd721a"
server_key_iv: "a20a7a5dccef46150d6c5ce08cf2debc"
server_key_version: "encryptionkey@0"
I20260927 07:06:57.166512 4268 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260927 07:06:57.170608 4283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:57.172070 4268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:06:57.172260 4268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "f68b78176da14edb9f10a1785ad804af"
format_stamp: "Formatted at 2026-09-27 07:06:57 on dist-test-slave-8vqf"
server_key: "60e2e0705ef90a27d0e354019fbd721a"
server_key_iv: "a20a7a5dccef46150d6c5ce08cf2debc"
server_key_version: "encryptionkey@0"
I20260927 07:06:57.172480 4268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:57.193982 4268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:57.198509 4268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:57.198880 4268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:57.199898 4268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:06:57.201207 4268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:06:57.201315 4268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:57.201462 4268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:06:57.201534 4268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:57.235718 4268 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:41273
I20260927 07:06:57.235812 4395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:41273 every 8 connection(s)
I20260927 07:06:57.237063 4268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:06:57.239563 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4268
I20260927 07:06:57.239744 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:06:57.240309 30785 external_mini_cluster.cc:1428] Setting key 4ac8ca5a74d3200dfac97e2bb5975830
I20260927 07:06:57.247534 4129 heartbeater.cc:503] Master 127.30.16.126:45209 was elected leader, sending a full tablet report...
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:06:57.258999 4131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.239820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:57931 (local address 127.30.16.126:45209)
0927 07:06:57.240138 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.240145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.241528 (+ 1383us) server_negotiation.cc:407] Connection header received
0927 07:06:57.243490 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.243496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.243597 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.243679 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.245313 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.246117 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.248133 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.248585 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.248739 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.252505 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.252511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.252515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.252550 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.256073 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.256791 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.256795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.256797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.256917 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.257352 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.257355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.257356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.257618 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.257724 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.258485 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.258738 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260927 07:06:57.260094 4398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.240531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:45209 (local address 127.30.16.67:57931)
0927 07:06:57.241353 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0927 07:06:57.241416 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 07:06:57.243239 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:06:57.243991 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:06:57.244006 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:06:57.244563 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:06:57.245175 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.245194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.246300 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.246305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.247976 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:06:57.247989 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.248835 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.248845 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:06:57.249191 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.249928 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:06:57.249960 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:06:57.252270 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:06:57.256202 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:57.256208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:57.256223 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:57.256676 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:57.257051 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:06:57.257054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:06:57.257057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:06:57.257249 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:06:57.257750 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:06:57.257756 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:06:57.258143 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 07:06:57.258384 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 07:06:57.258922 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":353,"threads_started":1}
I20260927 07:06:57.261390 4396 heartbeater.cc:347] Connected to a master server at 127.30.16.126:45209
I20260927 07:06:57.261714 4396 heartbeater.cc:464] Registering TS with master...
I20260927 07:06:57.262435 4396 heartbeater.cc:511] Master 127.30.16.126:45209 requested a full tablet report, sending...
I20260927 07:06:57.263911 3940 ts_manager.cc:194] Registered new tserver with Master: f68b78176da14edb9f10a1785ad804af (127.30.16.67:41273)
I20260927 07:06:57.264678 3940 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:57931
I20260927 07:06:57.268389 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/test-admin.keytab.
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.296931 30785 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 07:06:57.302455 4403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:57.299939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.30.16.65:45769)
0927 07:06:57.300623 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.300630 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.300658 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:06:57.300720 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.300726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.300940 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.301072 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.301989 (+ 917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45586: BlockingRecv error: recv got EOF from 127.0.0.1:45586 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":374,"threads_started":1}
W20260927 07:06:57.302706 30785 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc' of type FILE
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.304520 30785 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.306093 30785 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:57.323602 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.307991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.30.16.65:45769)
0927 07:06:57.308260 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.308266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.308531 (+ 265us) server_negotiation.cc:407] Connection header received
0927 07:06:57.308758 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.308762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.308828 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.308948 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.309993 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.311062 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.312397 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.313025 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.313162 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.317215 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.317230 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.317247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.317287 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.320576 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.321314 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.321324 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.321328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.321466 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.321879 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.321885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.321892 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.322286 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.322633 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.323048 (+ 415us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.323316 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 07:06:57.330314 4403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:57.328170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.30.16.65:45769)
0927 07:06:57.328490 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.328496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.328802 (+ 306us) server_negotiation.cc:407] Connection header received
0927 07:06:57.329019 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.329025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.329108 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.329211 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.330051 (+ 840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45602: BlockingRecv error: recv got EOF from 127.0.0.1:45602 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 07:06:57.330602 30785 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc' of type FILE
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.332247 30785 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.333580 30785 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:57.348508 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.334992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45606 (local address 127.30.16.65:45769)
0927 07:06:57.335225 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.335229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.335626 (+ 397us) server_negotiation.cc:407] Connection header received
0927 07:06:57.335759 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.335763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.335821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.335904 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.336880 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.337637 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.339043 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.339472 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.339573 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.343219 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.343224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.343228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.343261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.346143 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.346856 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.346860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.346863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.347040 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.347368 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.347372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.347375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.347644 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.347835 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.348115 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.348275 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260927 07:06:57.353536 4403 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:06:57.351555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.30.16.65:45769)
0927 07:06:57.351743 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.351751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.352067 (+ 316us) server_negotiation.cc:407] Connection header received
0927 07:06:57.352277 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.352282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.352359 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.352461 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.353294 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45608: BlockingRecv error: recv got EOF from 127.0.0.1:45608 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 07:06:57.354105 30785 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790492767955610-30785-0/krb5kdc/krb5cc)
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.356076 30785 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:57.357358 30785 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:06:57.373718 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.358694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.30.16.65:45769)
0927 07:06:57.358938 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.358956 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.359363 (+ 407us) server_negotiation.cc:407] Connection header received
0927 07:06:57.359539 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.359543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.359603 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.359686 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.360746 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.361904 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.363481 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.363959 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.364075 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.367988 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.367994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.367997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.368035 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.370902 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.371659 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.371664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.371667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.371832 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.372223 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.372226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.372228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.372516 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.372749 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.373146 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.373398 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:06:57.375662 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4001
I20260927 07:06:57.388749 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.376516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45626 (local address 127.30.16.65:45769)
0927 07:06:57.376740 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.376744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.377088 (+ 344us) server_negotiation.cc:407] Connection header received
0927 07:06:57.377292 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.377297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.377387 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.377465 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.378472 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.379302 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.380515 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.381005 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.381109 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.383248 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.383253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.383257 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.383289 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.386200 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.386982 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.386986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.386988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.387128 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.387447 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.387451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.387453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.387713 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.387927 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.388275 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.388467 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 07:06:57.389451 4124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:57.474875 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4001
I20260927 07:06:57.493075 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4135
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:06:57.509258 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.494036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.30.16.66:34207)
0927 07:06:57.494666 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.494673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.494699 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:06:57.494767 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.494773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.494992 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.495130 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.496087 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.497082 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.498341 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.498981 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.499084 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.503273 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.503285 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.503300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.503351 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.506317 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.507197 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.507206 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.507208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.507371 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.507772 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.507776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.507778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.508163 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.508453 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.508728 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.508946 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":341,"threads_started":1}
I20260927 07:06:57.510720 4258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:57.595229 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4135
I20260927 07:06:57.614109 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4268
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[3895](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:06:57.638747 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:57.615363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.30.16.67:41273)
0927 07:06:57.616094 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:57.616101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:57.616130 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:06:57.616191 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:57.616197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:57.616408 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:57.616549 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:57.617654 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.618956 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.620235 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:57.620821 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:57.620928 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:57.631996 (+ 11068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:57.632015 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:57.632031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:57.632083 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:57.635955 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.636719 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.636727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.636729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.636863 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:57.637277 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:57.637286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:57.637288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:57.637634 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:57.637900 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:57.638161 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 07:06:57.638379 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":331,"threads_started":1}
I20260927 07:06:57.640398 4391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:57.724788 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4268
I20260927 07:06:57.740862 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 3908
I20260927 07:06:57.742193 3970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:06:57.845824 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 3908
2026-09-27T07:06:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4034 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[4423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[4423](info): set up 2 sockets
Sep 27 07:06:57 dist-test-slave-8vqf krb5kdc[4423](info): commencing operation
krb5kdc: starting...
W20260927 07:06:58.947484 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:06:59 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:06:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab.
Entry for principal oryx/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab.
I20260927 07:06:59.165760 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39821
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab
--principal=oryx/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:06:59.502864 4436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:06:59.503168 4436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:06:59.503257 4436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:06:59.510615 4436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:06:59.510756 4436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:06:59.510807 4436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:06:59.510864 4436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:06:59.510908 4436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:06:59.520856 4436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab
--principal=oryx/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:06:59.522266 4436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:06:59.523610 4436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:06:59.535619 4441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.535694 4444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:06:59.535653 4442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:06:59.536654 4436 server_base.cc:1034] running on GCE node
I20260927 07:06:59.537497 4436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:06:59.539005 4436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:06:59.540215 4436 hybrid_clock.cc:648] HybridClock initialized: now 1790492819540146 us; error 54 us; skew 500 ppm
I20260927 07:06:59.540586 4436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:06:59 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:06:59.543840 4436 init.cc:377] Logged in from keytab as oryx/127.30.16.126@KRBTEST.COM (short username oryx)
I20260927 07:06:59.545341 4436 webserver.cc:460] Webserver started at http://127.30.16.126:40435/ using document root <none> and password file <none>
I20260927 07:06:59.546073 4436 fs_manager.cc:359] Metadata directory not provided
I20260927 07:06:59.546173 4436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:06:59.546422 4436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:06:59.549578 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "5a043bcea9144a32a230edcf5cdb242b"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-8vqf"
server_key: "e00e11cc112154ce006e23c94297ee47"
server_key_iv: "b86de598a028a73deca37011f220363e"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.550371 4436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "5a043bcea9144a32a230edcf5cdb242b"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-8vqf"
server_key: "e00e11cc112154ce006e23c94297ee47"
server_key_iv: "b86de598a028a73deca37011f220363e"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.555867 4436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:06:59.559616 4451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:06:59.561530 4436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:06:59.561734 4436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5a043bcea9144a32a230edcf5cdb242b"
format_stamp: "Formatted at 2026-09-27 07:06:59 on dist-test-slave-8vqf"
server_key: "e00e11cc112154ce006e23c94297ee47"
server_key_iv: "b86de598a028a73deca37011f220363e"
server_key_version: "encryptionkey@0"
I20260927 07:06:59.561951 4436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:06:59.576644 4436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:06:59.581072 4436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:06:59.581367 4436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:06:59.600972 4436 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:39821
I20260927 07:06:59.600982 4502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:39821 every 8 connection(s)
I20260927 07:06:59.602478 4436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:06:59.606432 4503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:06:59.607088 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4436
I20260927 07:06:59.607306 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:06:59.607980 30785 external_mini_cluster.cc:1428] Setting key ca243be63b0b7ee42a4409e368bdc46d
I20260927 07:06:59.621022 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Bootstrap starting.
Sep 27 07:06:59 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:06:59.624306 4503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Neither blocks nor log segments found. Creating new log.
I20260927 07:06:59.625747 4503 log.cc:824] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:06:59.629261 4503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: No bootstrap required, opened a new log
I20260927 07:06:59.633279 4506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:06:59.610007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.30.16.126:39821)
0927 07:06:59.610783 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 07:06:59.610809 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 07:06:59.610859 (+ 50us) server_negotiation.cc:407] Connection header received
0927 07:06:59.613172 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:06:59.613203 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:06:59.613750 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:06:59.614240 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:06:59.615776 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.617814 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.619220 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:06:59.620245 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:06:59.620427 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:06:59.624339 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:06:59.624350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:06:59.624371 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:06:59.624419 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:06:59.628382 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.629203 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.629213 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.629223 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.629437 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:06:59.629863 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:06:59.629868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:06:59.629872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:06:59.630611 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:06:59.630939 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 07:06:59.631325 (+ 386us) server_negotiation.cc:299] Negotiation successful
0927 07:06:59.631800 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":299,"threads_started":1}
I20260927 07:06:59.638761 4503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } }
I20260927 07:06:59.639104 4503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:06:59.639182 4503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a043bcea9144a32a230edcf5cdb242b, State: Initialized, Role: FOLLOWER
I20260927 07:06:59.639714 4503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } }
I20260927 07:06:59.639914 4503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:06:59.640010 4503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:06:59.640126 4503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:06:59.642524 4503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } }
I20260927 07:06:59.643042 4503 leader_election.cc:302] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a043bcea9144a32a230edcf5cdb242b; no voters:
I20260927 07:06:59.643599 4503 leader_election.cc:288] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:06:59.643731 4508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:06:59.644547 4508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 LEADER]: Becoming Leader. State: Replica: 5a043bcea9144a32a230edcf5cdb242b, State: Running, Role: LEADER
I20260927 07:06:59.645284 4508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } }
I20260927 07:06:59.645771 4503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:06:59.648675 4510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a043bcea9144a32a230edcf5cdb242b. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } }
I20260927 07:06:59.648639 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } }
I20260927 07:06:59.649205 4510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:59.649205 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:06:59.652984 4518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:06:59.659605 4518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:06:59.668286 4518 catalog_manager.cc:1347] Generated new cluster ID: aedf900fd14c47578e5a439c434c5244
I20260927 07:06:59.668419 4518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:06:59.679554 4518 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:06:59.681592 4518 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:06:59.693579 4518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Generated new TSK 0
I20260927 07:06:59.694814 4518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
I20260927 07:06:59.755224 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:00.105142 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:00.105517 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:00.105683 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:00.113215 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:00.113394 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:00.113595 4529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:00.123683 4529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:00.125094 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:00.126487 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:00.140903 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:00.140983 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:00.140937 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:00.141798 4529 server_base.cc:1034] running on GCE node
I20260927 07:07:00.142499 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:00.143572 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:00.144806 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790492820144726 us; error 77 us; skew 500 ppm
I20260927 07:07:00.145190 4529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:00 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:00.148730 4529 init.cc:377] Logged in from keytab as oryx/127.30.16.65@KRBTEST.COM (short username oryx)
I20260927 07:07:00.150331 4529 webserver.cc:460] Webserver started at http://127.30.16.65:42345/ using document root <none> and password file <none>
I20260927 07:07:00.151105 4529 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:00.151252 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:00.151561 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:00.154513 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "3f3a6d8b860944a79548ab3b133a9c75"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "500553336eb737c82782600648b80edb"
server_key_iv: "3b754c2dd5db6e1b084eb29ac5529a22"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.155416 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "3f3a6d8b860944a79548ab3b133a9c75"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "500553336eb737c82782600648b80edb"
server_key_iv: "3b754c2dd5db6e1b084eb29ac5529a22"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.161036 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:07:00.164815 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.166392 4529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:07:00.166632 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "3f3a6d8b860944a79548ab3b133a9c75"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "500553336eb737c82782600648b80edb"
server_key_iv: "3b754c2dd5db6e1b084eb29ac5529a22"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.166878 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:00.186831 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:00.191521 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:00.191831 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:00.192868 4529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:00.194386 4529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:00.194489 4529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.194662 4529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:00.194738 4529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.228016 4529 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41273
I20260927 07:07:00.228111 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41273 every 8 connection(s)
I20260927 07:07:00.229377 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:00.229933 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4529
I20260927 07:07:00.230177 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:00.231079 30785 external_mini_cluster.cc:1428] Setting key 7a2f7919449d1de20da84a2c629224f1
Sep 27 07:07:00 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:07:00.252461 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.232234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60063 (local address 127.30.16.126:39821)
0927 07:07:00.232710 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.232716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.234016 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 07:07:00.236952 (+ 2936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.236957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.237034 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.237128 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.239408 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.240184 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.241873 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.242277 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.242417 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.245972 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.245977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.245981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.246040 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.249109 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.250030 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.250036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.250039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.250168 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.250682 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.250687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.250696 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.250978 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.251133 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.251864 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.252084 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":218,"threads_started":1}
I20260927 07:07:00.253494 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.232990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:60063)
0927 07:07:00.233819 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.233868 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.236652 (+ 2784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.237415 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.237430 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.238256 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.239198 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.239226 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.240392 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.240396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.241664 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.241677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.242530 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.242537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.242807 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.243517 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.243539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.245699 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.249263 (+ 3564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.249273 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.249292 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.249895 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.250285 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.250293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.250298 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.250486 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.251108 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.251116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.251491 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.251764 (+ 273us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.252324 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.66@KRBTEST.COM" created.
I20260927 07:07:00.255276 4657 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:00.255597 4657 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:00.256381 4657 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
I20260927 07:07:00.258764 4468 ts_manager.cc:194] Registered new tserver with Master: 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273)
I20260927 07:07:00.260905 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:60063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
I20260927 07:07:00.282334 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:00.619000 4663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:00.619297 4663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:00.619470 4663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:00.626989 4663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:00.627106 4663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:00.627290 4663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:00.637266 4663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:00.638643 4663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:00.639997 4663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:00.653776 4668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:00.653849 4669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:00.653873 4671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:00.655277 4663 server_base.cc:1034] running on GCE node
I20260927 07:07:00.655961 4663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:00.657069 4663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:00.658303 4663 hybrid_clock.cc:648] HybridClock initialized: now 1790492820658231 us; error 63 us; skew 500 ppm
I20260927 07:07:00.658692 4663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:00 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:00.661969 4663 init.cc:377] Logged in from keytab as oryx/127.30.16.66@KRBTEST.COM (short username oryx)
I20260927 07:07:00.663586 4663 webserver.cc:460] Webserver started at http://127.30.16.66:35535/ using document root <none> and password file <none>
I20260927 07:07:00.664322 4663 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:00.664472 4663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:00.664780 4663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:00.667961 4663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "c28b93149fc84515a5645a9697ef73bb"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "37cbdf2c814168c6d99185159002bcec"
server_key_iv: "18fc21c48d1763ddab0d8b3ecfd22fd0"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.668787 4663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "c28b93149fc84515a5645a9697ef73bb"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "37cbdf2c814168c6d99185159002bcec"
server_key_iv: "18fc21c48d1763ddab0d8b3ecfd22fd0"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.674396 4663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 07:07:00.678048 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.679499 4663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:00.679682 4663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "c28b93149fc84515a5645a9697ef73bb"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "37cbdf2c814168c6d99185159002bcec"
server_key_iv: "18fc21c48d1763ddab0d8b3ecfd22fd0"
server_key_version: "encryptionkey@0"
I20260927 07:07:00.679901 4663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:00.700584 4663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:00.704826 4663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:00.705148 4663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:00.706141 4663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:00.707551 4663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:00.707659 4663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.707818 4663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:00.707895 4663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:00.742012 4663 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:43105
I20260927 07:07:00.742063 4790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:43105 every 8 connection(s)
I20260927 07:07:00.743496 4663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:00.743999 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4663
I20260927 07:07:00.744217 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:00.744854 30785 external_mini_cluster.cc:1428] Setting key 1de1f506ab6b42ecf3bbaf3fba2896c6
Sep 27 07:07:00 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492820, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:07:00.766516 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.746638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:40327 (local address 127.30.16.126:39821)
0927 07:07:00.746928 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:00.746935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:00.748418 (+ 1483us) server_negotiation.cc:407] Connection header received
0927 07:07:00.750450 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:00.750455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:00.750587 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:00.750701 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:00.752231 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.753195 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.755579 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.756052 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.756159 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.760541 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:00.760547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:00.760551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:00.760586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:00.763301 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.764044 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.764056 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.764059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.764187 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:00.764622 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:00.764631 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:00.764633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:00.764904 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:00.765058 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:00.765881 (+ 823us) server_negotiation.cc:299] Negotiation successful
0927 07:07:00.766110 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:07:00.767751 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:00.747355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:40327)
0927 07:07:00.748265 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:00.748310 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:00.750175 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:00.750965 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:00.750974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:00.751510 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:00.752069 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.752085 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.753367 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.753371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.755431 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:00.755449 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:00.756298 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:00.756309 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:00.756639 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:00.757437 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:00.757478 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:00.760283 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:00.763453 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.763463 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.763481 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.763914 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.764296 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:00.764310 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:00.764313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:00.764503 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:00.765030 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:00.765038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:00.765491 (+ 453us) client_negotiation.cc:769] Sending connection context
0927 07:07:00.765802 (+ 311us) client_negotiation.cc:240] Negotiation successful
0927 07:07:00.766374 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":379,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.67@KRBTEST.COM" created.
I20260927 07:07:00.769366 4791 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:00.769693 4791 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:00.770467 4791 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
I20260927 07:07:00.771948 4468 ts_manager.cc:194] Registered new tserver with Master: c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105)
I20260927 07:07:00.772838 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:40327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
I20260927 07:07:00.796231 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:01.130856 4796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:01.131160 4796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:01.131359 4796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:01.139040 4796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:01.139155 4796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:01.139364 4796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:01.149425 4796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:39821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:01.150869 4796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:01.152302 4796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:01.166541 4802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:01.166561 4801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:01.166648 4804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:01.167729 4796 server_base.cc:1034] running on GCE node
I20260927 07:07:01.168349 4796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:01.169404 4796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:01.170653 4796 hybrid_clock.cc:648] HybridClock initialized: now 1790492821170581 us; error 64 us; skew 500 ppm
I20260927 07:07:01.171016 4796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492821, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:01.174309 4796 init.cc:377] Logged in from keytab as oryx/127.30.16.67@KRBTEST.COM (short username oryx)
I20260927 07:07:01.176074 4796 webserver.cc:460] Webserver started at http://127.30.16.67:41961/ using document root <none> and password file <none>
I20260927 07:07:01.176777 4796 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:01.176919 4796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:01.177245 4796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:01.180328 4796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "f96b510a824d44899a323c0956d721cd"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "a1c0de4b80c91a5129bdc37f22f70037"
server_key_iv: "0cf384853a40f642d6c5a056f61408f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.181118 4796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "f96b510a824d44899a323c0956d721cd"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "a1c0de4b80c91a5129bdc37f22f70037"
server_key_iv: "0cf384853a40f642d6c5a056f61408f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.186901 4796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:07:01.190660 4811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:01.192014 4796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:07:01.192194 4796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "f96b510a824d44899a323c0956d721cd"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "a1c0de4b80c91a5129bdc37f22f70037"
server_key_iv: "0cf384853a40f642d6c5a056f61408f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.192396 4796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:01.207664 4796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:01.212172 4796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:01.212483 4796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:01.213546 4796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:01.214972 4796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:01.215071 4796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:01.215219 4796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:01.215289 4796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:01.250955 4796 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:35957
I20260927 07:07:01.251006 4923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:35957 every 8 connection(s)
I20260927 07:07:01.252482 4796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:01.257766 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4796
I20260927 07:07:01.258018 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:01.258626 30785 external_mini_cluster.cc:1428] Setting key 8beaf461aae3307b0397e95508dd2a1d
I20260927 07:07:01.264433 4657 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492821, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:01.275602 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.255450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:40245 (local address 127.30.16.126:39821)
0927 07:07:01.255684 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.255689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.256944 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 07:07:01.259768 (+ 2824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.259772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.259841 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.259928 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.262115 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.263001 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.264941 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.265412 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.265525 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.269382 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.269387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.269390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.269422 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.272244 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.273115 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.273119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.273122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.273284 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.273780 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.273785 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.273788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.274116 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.274254 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.275173 (+ 919us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.275336 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 07:07:01.276721 4926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.256007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:40245)
0927 07:07:01.256774 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.256820 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.259181 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.260281 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.260291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.261029 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.261890 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.261919 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.263180 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.263184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.264788 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.264801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.265640 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.265647 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.266017 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.266900 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.266924 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.269192 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.272428 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.272436 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.272450 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.273015 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.273422 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.273425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.273427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.273643 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.274258 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.274269 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.274806 (+ 537us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.275075 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.275496 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":300,"threads_started":1}
I20260927 07:07:01.278141 4924 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:01.278426 4924 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:01.279259 4924 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
I20260927 07:07:01.280673 4468 ts_manager.cc:194] Registered new tserver with Master: f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:01.281503 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:40245
I20260927 07:07:01.286813 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.16.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.125.keytab.
Entry for principal oryx/127.30.16.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.125.keytab.
I20260927 07:07:01.338752 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:39935
--webserver_interface=127.30.16.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.125.keytab
--principal=oryx/127.30.16.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39821,127.30.16.125:39935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:01.680265 4929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:01.680569 4929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:01.680754 4929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:01.688365 4929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:01.688482 4929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:01.688580 4929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:01.688680 4929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:01.688760 4929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:01.699815 4929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39821,127.30.16.125:39935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.125.keytab
--principal=oryx/127.30.16.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:39935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.16.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
W20260927 07:07:01.700233 4929 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.30.16.126:39821,127.30.16.125:39935'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 07:07:01.701347 4929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:01.702804 4929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:01.714886 4934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:01.714906 4935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:01.715237 4937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:01.715782 4929 server_base.cc:1034] running on GCE node
I20260927 07:07:01.716625 4929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:01.718176 4929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:01.719425 4929 hybrid_clock.cc:648] HybridClock initialized: now 1790492821719354 us; error 41 us; skew 500 ppm
I20260927 07:07:01.719764 4929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492821, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:01.722931 4929 init.cc:377] Logged in from keytab as oryx/127.30.16.125@KRBTEST.COM (short username oryx)
I20260927 07:07:01.724498 4929 webserver.cc:460] Webserver started at http://127.30.16.125:34571/ using document root <none> and password file <none>
I20260927 07:07:01.725171 4929 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:01.725258 4929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:01.725522 4929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:01.728578 4929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/instance:
uuid: "5e7413d90bde44999e00ce9fa6053b41"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "12b78af9125b78945c34a37c395b30d5"
server_key_iv: "820be8c697a5d90897ee100593410629"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.729346 4929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal/instance:
uuid: "5e7413d90bde44999e00ce9fa6053b41"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "12b78af9125b78945c34a37c395b30d5"
server_key_iv: "820be8c697a5d90897ee100593410629"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.734809 4929 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 07:07:01.738476 4944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:01.739930 4929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:01.740099 4929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "5e7413d90bde44999e00ce9fa6053b41"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "12b78af9125b78945c34a37c395b30d5"
server_key_iv: "820be8c697a5d90897ee100593410629"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.740316 4929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:01.756516 4929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:01.761168 4929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:01.761454 4929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:01.775600 4791 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
I20260927 07:07:01.781759 4929 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.125:39935
I20260927 07:07:01.781764 4995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.125:39935 every 8 connection(s)
I20260927 07:07:01.783268 4929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
I20260927 07:07:01.787472 4996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:01.790324 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 4929
I20260927 07:07:01.794946 4996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492821, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.125@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:01.815296 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.797765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45716 (local address 127.30.16.126:39821)
0927 07:07:01.798893 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.798900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.799488 (+ 588us) server_negotiation.cc:407] Connection header received
0927 07:07:01.799700 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.799704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.799765 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.799874 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.801258 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.802210 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.804192 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.804591 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.804691 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.808292 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.808301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.808317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.808386 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.811690 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.812654 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.812660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.812663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.812799 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.813258 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.813262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.813265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.813683 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.813837 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.814694 (+ 857us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.814935 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":998,"thread_start_us":850,"threads_started":1}
I20260927 07:07:01.816385 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.792628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.30.16.125:39935)
0927 07:07:01.793416 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.793428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.793468 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:07:01.795192 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.795216 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.795725 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.796712 (+ 987us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.797937 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.800171 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.801755 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.802749 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.802968 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.807719 (+ 4751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.807728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.807741 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.807776 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.811441 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.812343 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.812350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.812352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.812514 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.812975 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.812978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.812980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.813682 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.813972 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.814443 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.815023 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":573,"thread_start_us":369,"threads_started":1}
I20260927 07:07:01.816411 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45716)
0927 07:07:01.799356 (+ 1617us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.799392 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.799570 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.799929 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.799936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.800774 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.801096 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.801107 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.802359 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.802362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.804039 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.804054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.804833 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.804843 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.805053 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.805688 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.805718 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.808095 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.811899 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.811910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.811924 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.812500 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.812921 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.812942 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.812948 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.813133 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.813862 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.813871 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.814151 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.814492 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.815163 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"mutex_wait_us":99,"thread_start_us":1293,"threads_started":1}
I20260927 07:07:01.818535 4996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492821, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.125@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
I20260927 07:07:01.832126 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.819627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33218)
0927 07:07:01.819855 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.819873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.819971 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.820266 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.820270 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.820651 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.820806 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.820815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.821791 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.821795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.822837 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.822849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.823554 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.823559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.823700 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.824285 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.824301 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.826475 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.829745 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.829749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.829751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.830127 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.830531 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.830534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.830537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.830727 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.831271 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.831275 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.831413 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.831558 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.831796 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 07:07:01.832202 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.819474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.30.16.125:39935)
0927 07:07:01.819731 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.819738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.819995 (+ 257us) server_negotiation.cc:407] Connection header received
0927 07:07:01.820065 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.820070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.820147 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.820230 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.820981 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.821662 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.822988 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.823414 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.823518 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.826751 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.826757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.826760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.826869 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.829597 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.830252 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.830256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.830258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.830386 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.830839 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.830843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.830846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.831148 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.831265 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.831657 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.831927 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":84}
I20260927 07:07:01.837266 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Bootstrap starting.
I20260927 07:07:01.840435 4996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:01.841481 4996 log.cc:824] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:01.844537 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: No bootstrap required, opened a new log
I20260927 07:07:01.851878 4996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } }
I20260927 07:07:01.852262 4996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:01.852371 4996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e7413d90bde44999e00ce9fa6053b41, State: Initialized, Role: FOLLOWER
I20260927 07:07:01.853050 4996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } }
I20260927 07:07:01.853943 5003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } } }
I20260927 07:07:01.854254 5003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:01.855090 4996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 07:07:01.864940 5014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:07:01.865155 5014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:07:01.867439 4929 master_runner.cc:428] Detected that this master 5e7413d90bde44999e00ce9fa6053b41 is joining an existing cluster
I20260927 07:07:01.867551 4929 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.16.125:39935
I20260927 07:07:01.897532 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.874792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45720 (local address 127.30.16.126:39821)
0927 07:07:01.875031 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.875040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.881649 (+ 6609us) server_negotiation.cc:407] Connection header received
0927 07:07:01.882318 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.882327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.882461 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.882583 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.884288 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.885056 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.887105 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.887521 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.887671 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.890035 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.890045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.890049 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.890113 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.893783 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.894706 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.894712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.894716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.894879 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.895337 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.895342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.895346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.895833 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.896006 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.896924 (+ 918us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.897161 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 07:07:01.898840 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.872067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45720)
0927 07:07:01.881507 (+ 9440us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.881533 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.881736 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.882845 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.882851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.883672 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.884122 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.884131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.885191 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.885197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.886969 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.886983 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.887723 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.887730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.887900 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.888584 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.888610 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.889776 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.893946 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.893951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.893956 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.894510 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.895021 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.895026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.895030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.895206 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.896069 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.896074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.896303 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.896472 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.898290 (+ 1818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9229,"mutex_wait_us":28,"thread_start_us":9748,"threads_started":1}
I20260927 07:07:01.899425 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.872176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33224)
0927 07:07:01.883744 (+ 11568us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.883769 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.883911 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.884525 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.884530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.885187 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.885654 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.885666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.887071 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.887076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.888555 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.888569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.889664 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.889671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.889843 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.890511 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.890530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.891478 (+ 948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.895298 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.895306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.895310 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.895984 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.896521 (+ 537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.896529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.896535 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.896721 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.897613 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.897618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.897756 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.898139 (+ 383us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.898878 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11432,"mutex_wait_us":32,"thread_start_us":11245,"threads_started":1}
I20260927 07:07:01.899817 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.872567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.30.16.125:39935)
0927 07:07:01.872750 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.872757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.883911 (+ 11154us) server_negotiation.cc:407] Connection header received
0927 07:07:01.884107 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.884112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.884184 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.884300 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.885820 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.886884 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.888700 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.889451 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.889593 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.891637 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.891643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.891647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.891683 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.895153 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.896117 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.896125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.896130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.896296 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.896927 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.896935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.896940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.897322 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.897526 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.898025 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.898636 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":347,"server-negotiator.queue_time_us":60,"spinlock_wait_cycles":52608}
WARNING: no policy specified for oryx/127.30.16.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.124@KRBTEST.COM" created.
Sep 27 07:07:01 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.126@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
I20260927 07:07:01.924103 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.909257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.30.16.125:39935)
0927 07:07:01.909459 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.909469 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.910112 (+ 643us) server_negotiation.cc:407] Connection header received
0927 07:07:01.910515 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.910525 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.910669 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.910796 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.911943 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.912648 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.913989 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.914478 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.914659 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.918027 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:01.918033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:01.918037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:01.918069 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:01.921055 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.921847 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.921853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.921857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.922039 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:01.922446 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:01.922453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:01.922456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:01.922817 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:01.923020 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:01.923484 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 07:07:01.923720 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:07:01.924150 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.909241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33238)
0927 07:07:01.909990 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.910025 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.910259 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.910853 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.910861 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.911566 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.911778 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.911788 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.912786 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.912791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.913838 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.913850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.914678 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.914685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.914891 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:01.915602 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:01.915620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:01.917808 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:01.921226 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.921240 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.921249 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.921691 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.922170 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:01.922175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:01.922179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:01.922328 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:01.922967 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:01.922976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:01.923177 (+ 201us) client_negotiation.cc:769] Sending connection context
0927 07:07:01.923376 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 07:07:01.923727 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":360,"threads_started":1}
I20260927 07:07:01.926508 4468 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.16.125:39935: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41"
member_type: NON_VOTER
last_known_addr {
host: "127.30.16.125"
port: 39935
}
attrs {
promote: true
}
}
dest_uuid: "5a043bcea9144a32a230edcf5cdb242b"
cas_config_opid_index: -1
I20260927 07:07:01.928102 4468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } }
I20260927 07:07:01.931965 5027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5e7413d90bde44999e00ce9fa6053b41 (127.30.16.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } }
I20260927 07:07:01.933297 4971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.124.keytab.
Entry for principal oryx/127.30.16.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.124.keytab.
I20260927 07:07:01.936463 5027 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.16.125:39935
I20260927 07:07:01.936925 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } }
I20260927 07:07:01.937043 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:46635
--webserver_interface=127.30.16.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.124.keytab
--principal=oryx/127.30.16.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:01.937121 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:01.939024 4929 master.cc:537] Master@127.30.16.125:39935 shutting down...
I20260927 07:07:01.941394 4971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 07:07:01.943048 5027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 07:07:01.947795 4455 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 07:07:01.950117 4455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b -> Peer 5e7413d90bde44999e00ce9fa6053b41 (127.30.16.125:39935): Couldn't send request to peer 5e7413d90bde44999e00ce9fa6053b41. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 07:07:01.953359 4929 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:01.953809 4929 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:01.954000 4929 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: stopping tablet replica
I20260927 07:07:01.960991 4929 master.cc:552] Master@127.30.16.125:39935 shutdown complete.
I20260927 07:07:01.961158 4929 master_runner.cc:305] Clearing existing system tablet
I20260927 07:07:01.961721 4929 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:01.961900 4929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:01.965564 4929 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:01.967619 5032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:01.968052 4929 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:01.968479 4929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 07:07:01.968611 4929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "5e7413d90bde44999e00ce9fa6053b41"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "12b78af9125b78945c34a37c395b30d5"
server_key_iv: "820be8c697a5d90897ee100593410629"
server_key_version: "encryptionkey@0"
I20260927 07:07:01.974067 4929 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:01.981514 4929 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 07:07:01.981827 4929 log.cc:1196] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 07:07:01.982504 4929 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Deleting consensus metadata
I20260927 07:07:01.982870 4929 master_runner.cc:315] Copying system tablet from 127.30.16.126:39821
I20260927 07:07:01.985558 4929 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: tablet copy: Beginning tablet copy session from remote peer at address 127.30.16.126:39821
I20260927 07:07:02.010318 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.986416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45730)
0927 07:07:01.995125 (+ 8709us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:01.995152 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:01.995367 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:01.995823 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:01.995827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:01.996404 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:01.996697 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.996734 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.998202 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.998211 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:01.999809 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:01.999823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.000836 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.000842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.001016 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.001795 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.001846 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.003021 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.007261 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.007267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.007282 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.007812 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.008279 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.008283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.008288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.008490 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.009145 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.009149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.009345 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.009517 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.009861 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8532,"thread_start_us":8351,"threads_started":1}
I20260927 07:07:02.011343 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:01.986475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.30.16.126:39821)
0927 07:07:01.986713 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:01.986719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:01.995319 (+ 8600us) server_negotiation.cc:407] Connection header received
0927 07:07:01.995535 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:01.995541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:01.995647 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:01.995754 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:01.996887 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:01.998037 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:01.999967 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.000663 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.000794 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.003262 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.003269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.003274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.003345 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.007076 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.007972 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.007978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.007982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.008141 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.008629 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.008635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.008638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.009007 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.009158 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.010740 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.010982 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260927 07:07:02.012341 4488 tablet_copy_service.cc:140] P 5a043bcea9144a32a230edcf5cdb242b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5e7413d90bde44999e00ce9fa6053b41 ({username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:45730)
I20260927 07:07:02.012631 4488 tablet_copy_service.cc:161] P 5a043bcea9144a32a230edcf5cdb242b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5e7413d90bde44999e00ce9fa6053b41 at {username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:45730: session id = 5e7413d90bde44999e00ce9fa6053b41-00000000000000000000000000000000
I20260927 07:07:02.017202 4488 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 07:07:02.019762 4929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 07:07:02.027238 4929 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: tablet copy: Starting download of 0 data blocks...
I20260927 07:07:02.027623 4929 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: tablet copy: Starting download of 1 WAL segments...
I20260927 07:07:02.030927 4929 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 07:07:02.035800 4488 tablet_copy_service.cc:342] P 5a043bcea9144a32a230edcf5cdb242b: Request end of tablet copy session 5e7413d90bde44999e00ce9fa6053b41-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:45730
I20260927 07:07:02.036051 4488 tablet_copy_service.cc:433] P 5a043bcea9144a32a230edcf5cdb242b: ending tablet copy session 5e7413d90bde44999e00ce9fa6053b41-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5e7413d90bde44999e00ce9fa6053b41
I20260927 07:07:02.048261 4929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:02.051184 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:02.056190 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.060734 4929 server_base.cc:1034] running on GCE node
W20260927 07:07:02.060901 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.061378 4929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:02.062084 4929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:02.063315 4929 hybrid_clock.cc:648] HybridClock initialized: now 1790492822063202 us; error 75 us; skew 500 ppm
I20260927 07:07:02.063570 4929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:02.065914 4929 init.cc:377] Logged in from keytab as oryx/127.30.16.125@KRBTEST.COM (short username oryx)
I20260927 07:07:02.071707 4929 webserver.cc:460] Webserver started at http://127.30.16.125:46409/ using document root <none> and password file <none>
I20260927 07:07:02.072157 4929 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:02.072364 4929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:02.075893 4929 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 07:07:02.077764 5049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:02.078212 4929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260927 07:07:02.078404 4929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "5e7413d90bde44999e00ce9fa6053b41"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "12b78af9125b78945c34a37c395b30d5"
server_key_iv: "820be8c697a5d90897ee100593410629"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.078676 4929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:02.105931 4929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:02.111025 4929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:02.137085 4929 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.125:39935
I20260927 07:07:02.137087 5100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.125:39935 every 8 connection(s)
I20260927 07:07:02.138372 4929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
I20260927 07:07:02.143872 5101 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:02.147651 5101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Bootstrap starting.
I20260927 07:07:02.165088 5101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:02.165671 5101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Bootstrap complete.
I20260927 07:07:02.166931 5101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } }
I20260927 07:07:02.167241 5101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e7413d90bde44999e00ce9fa6053b41, State: Initialized, Role: LEARNER
I20260927 07:07:02.167568 5101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } }
I20260927 07:07:02.168828 5101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:02.170970 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } }
I20260927 07:07:02.171377 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: LEARNER
I20260927 07:07:02.194530 5115 catalog_manager.cc:1259] Loaded cluster ID: aedf900fd14c47578e5a439c434c5244
I20260927 07:07:02.194695 5115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: loading cluster ID for follower catalog manager: success
I20260927 07:07:02.199191 5115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: acquiring CA information for follower catalog manager: success
I20260927 07:07:02.201938 5115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:07:02.284469 4924 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
W20260927 07:07:02.359766 5029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:02.360136 5029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:02.360287 5029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:02.368366 5029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:02.368578 5029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:02.368682 5029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:02.368788 5029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:02.368870 5029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:02.380903 5029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.124.keytab
--principal=oryx/127.30.16.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:46635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.16.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:02.382473 5029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:02.384315 5029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 07:07:02.395041 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.388278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33254)
0927 07:07:02.388498 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.388527 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.388718 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.389485 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.389490 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.389519 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:02.389721 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.389731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.391596 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.391604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.392888 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.392901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.394106 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.394111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.394374 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.394462 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.394598 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.394758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 07:07:02.395334 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.388481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33254 (local address 127.30.16.125:39935)
0927 07:07:02.389146 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.389150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.389166 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:02.389224 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.389227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.389302 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.389445 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:02.389899 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.391390 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.393059 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.393903 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.394056 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.394315 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.394828 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.394983 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":412,"threads_started":1}
I20260927 07:07:02.396438 5076 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 07:07:02.397054 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a043bcea9144a32a230edcf5cdb242b. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } }
I20260927 07:07:02.397339 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: LEARNER
W20260927 07:07:02.401747 5121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:02.401808 5122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:02.401891 5124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.402470 5029 server_base.cc:1034] running on GCE node
I20260927 07:07:02.403429 5029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:02.404987 5029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:02.406227 5029 hybrid_clock.cc:648] HybridClock initialized: now 1790492822406168 us; error 42 us; skew 500 ppm
I20260927 07:07:02.406770 5029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:02.410120 5029 init.cc:377] Logged in from keytab as oryx/127.30.16.124@KRBTEST.COM (short username oryx)
I20260927 07:07:02.411901 5029 webserver.cc:460] Webserver started at http://127.30.16.124:42579/ using document root <none> and password file <none>
I20260927 07:07:02.412636 5029 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:02.412781 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:02.413100 5029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:02.416379 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/instance:
uuid: "49632a44d46b4a7f8062296b9ba3ada5"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-8vqf"
server_key: "a97ac3b1fba7684798f194d9ea625834"
server_key_iv: "4da03c403eb0e3f892518b2de1e73877"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.417161 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal/instance:
uuid: "49632a44d46b4a7f8062296b9ba3ada5"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-8vqf"
server_key: "a97ac3b1fba7684798f194d9ea625834"
server_key_iv: "4da03c403eb0e3f892518b2de1e73877"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.422708 5029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260927 07:07:02.426265 5131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:02.427824 5029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:02.428036 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "49632a44d46b4a7f8062296b9ba3ada5"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-8vqf"
server_key: "a97ac3b1fba7684798f194d9ea625834"
server_key_iv: "4da03c403eb0e3f892518b2de1e73877"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.428259 5029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:02.457587 5029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:02.462353 5029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:02.462725 5029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:02.481753 5029 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.124:46635
I20260927 07:07:02.481771 5182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.124:46635 every 8 connection(s)
I20260927 07:07:02.483275 5029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
I20260927 07:07:02.487619 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:02.489320 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5029
I20260927 07:07:02.498762 5183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.124@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:02.515846 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.492055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.30.16.124:46635)
0927 07:07:02.492922 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.492939 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.493002 (+ 63us) server_negotiation.cc:407] Connection header received
0927 07:07:02.495110 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.495145 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.495749 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.496350 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.497844 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.499997 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.501388 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.502393 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.502665 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.507037 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.507046 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.507058 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.507101 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.510782 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.511732 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.511741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.511750 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.511957 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.512389 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.512393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.512400 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.513125 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.513434 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.513946 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.514453 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":355,"threads_started":1}
I20260927 07:07:02.518239 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.502009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.30.16.126:39821)
0927 07:07:02.502251 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.502259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.503086 (+ 827us) server_negotiation.cc:407] Connection header received
0927 07:07:02.503320 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.503333 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.503403 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.503518 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.504664 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.505378 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.506962 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.507381 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.507479 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.511747 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.511754 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.511760 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.511801 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.515192 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.515972 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.515977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.515981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.516122 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.516565 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.516569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.516573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.516909 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.517057 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.517569 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.517850 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 07:07:02.518347 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.501970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45742)
0927 07:07:02.502943 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.502988 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.503174 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.503578 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.503585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.504233 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.504499 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.504508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.505528 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.505537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.506790 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.506803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.507591 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.507600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.507783 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.508412 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.508439 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.511532 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.515353 (+ 3821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.515363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.515368 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.515823 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.516251 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.516254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.516257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.516435 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.517037 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.517043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.517200 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.517459 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.517809 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":517,"threads_started":1}
I20260927 07:07:02.519733 5183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.124@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
I20260927 07:07:02.533404 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.520665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.30.16.125:39935)
0927 07:07:02.520881 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.520889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.520909 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:07:02.520975 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.520979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.521038 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.521151 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.521844 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.522691 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.524062 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.524526 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.524669 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.528012 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.528025 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.528030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.528064 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.530775 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.531445 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.531452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.531455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.531617 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.532002 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.532007 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.532010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.532293 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.532456 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.532803 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.533048 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:07:02.533404 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.520473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33262)
0927 07:07:02.520656 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.520673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.520796 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.521169 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.521174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.521538 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.521685 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.521694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.522946 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.522959 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.523892 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.523903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.524704 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.524709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.524884 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.525523 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.525540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.527752 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.530922 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.530928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.530932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.531314 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.531732 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.531736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.531740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.531883 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.532413 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.532418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.532575 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.532736 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.533048 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 07:07:02.534840 5183 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.124@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
I20260927 07:07:02.550165 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.535597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54488)
0927 07:07:02.535836 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.535855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.535953 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.536439 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.536443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.536994 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.537223 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.537234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.538646 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.538656 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.540027 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.540039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.540976 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.540982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.541183 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.541806 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.541829 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.544464 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.547510 (+ 3046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.547514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.547516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.547962 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.548419 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.548422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.548425 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.548568 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.549288 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.549292 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.549430 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.549551 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.549823 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 07:07:02.550472 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.535761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54488 (local address 127.30.16.124:46635)
0927 07:07:02.535918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.535923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.535939 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:02.536074 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.536079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.536142 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.536239 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.537415 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.538452 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.540159 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.540799 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.540941 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.544645 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.544650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.544653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.544682 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.547382 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.548136 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.548144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.548149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.548306 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.548706 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.548714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.548718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.549067 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.549177 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.549688 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.550103 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":193,"server-negotiator.queue_time_us":54}
I20260927 07:07:02.555752 5183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Bootstrap starting.
I20260927 07:07:02.559228 5183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:02.560364 5183 log.cc:824] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:02.563616 5183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: No bootstrap required, opened a new log
I20260927 07:07:02.571657 5183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } }
I20260927 07:07:02.572046 5183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:02.572199 5183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49632a44d46b4a7f8062296b9ba3ada5, State: Initialized, Role: FOLLOWER
I20260927 07:07:02.572827 5183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } }
I20260927 07:07:02.573715 5189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } } }
I20260927 07:07:02.574021 5189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:02.574949 5183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 07:07:02.585193 5200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:07:02.585340 5200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:07:02.589442 5029 master_runner.cc:428] Detected that this master 49632a44d46b4a7f8062296b9ba3ada5 is joining an existing cluster
I20260927 07:07:02.589555 5029 master_runner.cc:432] Initiating AddMaster RPC to add 127.30.16.124:46635
I20260927 07:07:02.612865 5119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.593973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.30.16.125:39935)
0927 07:07:02.594187 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.594196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.595785 (+ 1589us) server_negotiation.cc:407] Connection header received
0927 07:07:02.596077 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.596083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.596168 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.596270 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.598109 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.599325 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.601495 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.602224 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.602327 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.605433 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.605443 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.605449 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.605504 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.609287 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.610214 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.610221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.610225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.610392 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.610932 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.610937 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.610940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.611306 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.611425 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.612141 (+ 716us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.612431 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:07:02.614248 5207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.594098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33270)
0927 07:07:02.595686 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.595703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.595895 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.596566 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.596570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.597457 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.597908 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.597920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.599532 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.599537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.601301 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.601317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.602747 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.602754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.602987 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.603835 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.603863 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.605185 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.609479 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.609485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.609490 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.610062 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.610575 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.610579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.610583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.610775 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.611529 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.611535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.611798 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.613136 (+ 1338us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.613549 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"mutex_wait_us":3,"thread_start_us":1089,"threads_started":1}
I20260927 07:07:02.622629 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4529
I20260927 07:07:02.623653 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.595681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54502 (local address 127.30.16.124:46635)
0927 07:07:02.595844 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.595850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.607170 (+ 11320us) server_negotiation.cc:407] Connection header received
0927 07:07:02.607352 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.607356 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.607419 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.607508 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.608828 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.609966 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.612080 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.612799 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.612896 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.616375 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.616382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.616387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.616432 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.620363 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.621158 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.621164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.621168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.621289 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.621710 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.621715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.621719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.622214 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.622365 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.623034 (+ 669us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.623274 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 07:07:02.625020 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.595830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54502)
0927 07:07:02.607053 (+ 11223us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.607075 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.607234 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.607669 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.607674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.608331 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.608674 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.608688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.610184 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.610189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.611925 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.611939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.613523 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.613532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.613712 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.615012 (+ 1300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.615034 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.616173 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.620521 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.620526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.620531 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.621042 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.621429 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.621433 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.621438 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.621598 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.622435 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.622441 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.622773 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.623842 (+ 1069us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.624217 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11068,"mutex_wait_us":13,"thread_start_us":10923,"threads_started":1}
I20260927 07:07:02.630405 5206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.594009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45752)
0927 07:07:02.594772 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.594795 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.595007 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.595664 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.595674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.596412 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.596817 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.596831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.598604 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.598610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.610947 (+ 12337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.610965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.612364 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.612372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.612611 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.613548 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.613578 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.614871 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.627012 (+ 12141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.627017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.627021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.627504 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.628136 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.628140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.628144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.628324 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.629227 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.629233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.629435 (+ 202us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.629599 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.629950 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":1,"thread_start_us":749,"threads_started":1}
I20260927 07:07:02.630494 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.594533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.30.16.126:39821)
0927 07:07:02.594752 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.594759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.594886 (+ 127us) server_negotiation.cc:407] Connection header received
0927 07:07:02.595201 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.595210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.595313 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.595417 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.597058 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.598345 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.611152 (+ 12807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.611911 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.612031 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.618747 (+ 6716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.618755 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.618760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.618804 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.626268 (+ 7464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.627658 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.627668 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.627672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.627988 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.628490 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.628499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.628506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.629063 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.629279 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.629836 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.630099 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.65@KRBTEST.COM
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.126@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
I20260927 07:07:02.655695 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.624023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49804 (local address 127.30.16.65:41273)
0927 07:07:02.631045 (+ 7022us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.631055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.631094 (+ 39us) server_negotiation.cc:407] Connection header received
0927 07:07:02.631154 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.631161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.631394 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.631598 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.633471 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.635105 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.636691 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.637631 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.637780 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.647492 (+ 9712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.647507 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.647523 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.647559 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.651231 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.652456 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.652469 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.652473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.652633 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.653105 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.653113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.653117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.653555 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.653861 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.654967 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.655244 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6819,"thread_start_us":9935,"threads_started":1}
I20260927 07:07:02.657094 4652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:02.662992 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.645724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54504)
0927 07:07:02.646010 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.646030 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.646145 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.646610 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.646615 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.647102 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.647311 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.647322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.648816 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.648823 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.650352 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.650367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.651476 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.651484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.651644 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.652233 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.652256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.654818 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.660190 (+ 5372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.660199 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.660206 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.660661 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.661273 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.661279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.661282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.661406 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.661942 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.661946 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.662109 (+ 163us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.662261 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.662627 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260927 07:07:02.662989 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.645801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.30.16.124:46635)
0927 07:07:02.645994 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.646003 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.646116 (+ 113us) server_negotiation.cc:407] Connection header received
0927 07:07:02.646318 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.646327 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.646425 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.646539 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.647487 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.648671 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.650502 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.651277 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.651443 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.655064 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.655070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.655074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.655111 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.659984 (+ 4873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.660800 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.660805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.660807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.660912 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.661533 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.661538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.661541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.661818 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.661982 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.662350 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.662617 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:07:02.664434 4468 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.30.16.124:46635: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5"
member_type: NON_VOTER
last_known_addr {
host: "127.30.16.124"
port: 46635
}
attrs {
promote: true
}
}
dest_uuid: "5a043bcea9144a32a230edcf5cdb242b"
cas_config_opid_index: 5
I20260927 07:07:02.665447 4468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } }
I20260927 07:07:02.667135 5027 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.667762 5076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 07:07:02.669046 5158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:02.670634 5027 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.30.16.124:46635
I20260927 07:07:02.685015 5211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 07:07:02.685720 5212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.686118 5212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:02.691329 5158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 07:07:02.692091 5029 master.cc:537] Master@127.30.16.124:46635 shutting down...
I20260927 07:07:02.692386 5027 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:02.704937 5029 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:02.705390 5029 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:02.705613 5029 tablet_replica.cc:330] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: stopping tablet replica
I20260927 07:07:02.716413 5076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.720675 5029 master.cc:552] Master@127.30.16.124:46635 shutdown complete.
I20260927 07:07:02.720778 5029 master_runner.cc:305] Clearing existing system tablet
I20260927 07:07:02.721171 5029 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:02.721287 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260927 07:07:02.726716 4455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b -> Peer 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124:46635): Couldn't send request to peer 49632a44d46b4a7f8062296b9ba3ada5. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 07:07:02.728137 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: NON_VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: true } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.728456 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: LEARNER
I20260927 07:07:02.729014 5212 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: attempting to promote NON_VOTER 5e7413d90bde44999e00ce9fa6053b41 to VOTER
I20260927 07:07:02.729755 5212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } }
I20260927 07:07:02.731835 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.731392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54520)
0927 07:07:02.731554 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.731677 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:46635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 07:07:02.732795 5076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 LEARNER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 07:07:02.732838 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.732453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54528)
0927 07:07:02.732603 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.732695 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:46635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 07:07:02.733045 4455 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b -> Peer 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124:46635): Couldn't send request to peer 49632a44d46b4a7f8062296b9ba3ada5. Status: Network error: Client connection negotiation failed: client connection to 127.30.16.124:46635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 07:07:02.733654 5212 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 07:07:02.734786 5029 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.004s sys 0.000s
I20260927 07:07:02.735919 5211 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e7413d90bde44999e00ce9fa6053b41 (127.30.16.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.748837 5076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e7413d90bde44999e00ce9fa6053b41 (127.30.16.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.755164 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.755575 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:02.756199 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.756486 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:02.760170 5217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:02.760692 5029 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:02.760848 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.761132 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:02.761317 5029 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.005s sys 0.009s
I20260927 07:07:02.761685 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "49632a44d46b4a7f8062296b9ba3ada5"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-8vqf"
server_key: "a97ac3b1fba7684798f194d9ea625834"
server_key_iv: "4da03c403eb0e3f892518b2de1e73877"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.766378 5029 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:02.772285 5029 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 07:07:02.772531 5029 log.cc:1196] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 07:07:02.772984 5029 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Deleting consensus metadata
I20260927 07:07:02.773228 5029 master_runner.cc:315] Copying system tablet from 127.30.16.126:39821
I20260927 07:07:02.776053 5029 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: tablet copy: Beginning tablet copy session from remote peer at address 127.30.16.126:39821
I20260927 07:07:02.795109 5000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.776915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.30.16.126:39821)
0927 07:07:02.777185 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:02.777193 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:02.783168 (+ 5975us) server_negotiation.cc:407] Connection header received
0927 07:07:02.783428 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:02.783436 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:02.783544 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:02.783638 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:02.784621 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.785369 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.786676 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.787138 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.787289 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.789510 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:02.789516 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:02.789529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:02.789562 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:02.792394 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.793131 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.793137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.793141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.793277 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:02.793659 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:02.793666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:02.793669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:02.793951 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:02.794118 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:02.794489 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 07:07:02.794742 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 07:07:02.795463 5224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:02.776900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:45760)
0927 07:07:02.783034 (+ 6134us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:02.783055 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:02.783280 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:02.783667 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:02.783671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:02.784122 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:02.784413 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.784424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.785525 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.785531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.786511 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:02.786523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:02.787300 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:02.787307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:02.787495 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:02.788325 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:02.788343 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:02.789312 (+ 969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:02.792566 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.792572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.792576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.793002 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.793389 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:02.793394 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:02.793402 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:02.793545 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:02.794067 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:02.794072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:02.794248 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 07:07:02.794391 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 07:07:02.794729 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5994,"thread_start_us":5851,"threads_started":1}
I20260927 07:07:02.796128 4488 tablet_copy_service.cc:140] P 5a043bcea9144a32a230edcf5cdb242b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 49632a44d46b4a7f8062296b9ba3ada5 ({username='oryx', principal='oryx/127.30.16.124@KRBTEST.COM'} at 127.0.0.1:45760)
I20260927 07:07:02.796303 4488 tablet_copy_service.cc:161] P 5a043bcea9144a32a230edcf5cdb242b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 49632a44d46b4a7f8062296b9ba3ada5 at {username='oryx', principal='oryx/127.30.16.124@KRBTEST.COM'} at 127.0.0.1:45760: session id = 49632a44d46b4a7f8062296b9ba3ada5-00000000000000000000000000000000
I20260927 07:07:02.798871 4488 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 07:07:02.800550 5029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 07:07:02.805290 5029 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: tablet copy: Starting download of 0 data blocks...
I20260927 07:07:02.805593 5029 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: tablet copy: Starting download of 1 WAL segments...
I20260927 07:07:02.807876 5029 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 07:07:02.811177 4488 tablet_copy_service.cc:342] P 5a043bcea9144a32a230edcf5cdb242b: Request end of tablet copy session 49632a44d46b4a7f8062296b9ba3ada5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.30.16.124@KRBTEST.COM'} at 127.0.0.1:45760
I20260927 07:07:02.811324 4488 tablet_copy_service.cc:433] P 5a043bcea9144a32a230edcf5cdb242b: ending tablet copy session 49632a44d46b4a7f8062296b9ba3ada5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 49632a44d46b4a7f8062296b9ba3ada5
I20260927 07:07:02.821369 5029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:02.824246 5225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:02.829701 5226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.830258 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4529
I20260927 07:07:02.842823 5029 server_base.cc:1034] running on GCE node
W20260927 07:07:02.843853 5228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:02.844398 5029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:02.845132 5029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:02.846349 5029 hybrid_clock.cc:648] HybridClock initialized: now 1790492822846305 us; error 45 us; skew 500 ppm
I20260927 07:07:02.846735 5029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:02 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492822, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:02.849125 5029 init.cc:377] Logged in from keytab as oryx/127.30.16.124@KRBTEST.COM (short username oryx)
I20260927 07:07:02.859768 5029 webserver.cc:460] Webserver started at http://127.30.16.124:44445/ using document root <none> and password file <none>
I20260927 07:07:02.860206 5029 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:02.860319 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:02.861675 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:41273
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:39821
--webserver_port=42345
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:02.865105 5029 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 07:07:02.867244 5235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:02.867702 5029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 07:07:02.867830 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "49632a44d46b4a7f8062296b9ba3ada5"
format_stamp: "Formatted at 2026-09-27 07:07:02 on dist-test-slave-8vqf"
server_key: "a97ac3b1fba7684798f194d9ea625834"
server_key_iv: "4da03c403eb0e3f892518b2de1e73877"
server_key_version: "encryptionkey@0"
I20260927 07:07:02.868029 5029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:02.886284 5029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:02.891091 5029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:02.930086 5029 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.124:46635
I20260927 07:07:02.930253 5287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.124:46635 every 8 connection(s)
I20260927 07:07:02.931294 5029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
I20260927 07:07:02.936903 5288 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:02.940825 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Bootstrap starting.
I20260927 07:07:02.961174 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:02.961723 5288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: Bootstrap complete.
I20260927 07:07:02.962895 5288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } }
I20260927 07:07:02.963238 5288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49632a44d46b4a7f8062296b9ba3ada5, State: Initialized, Role: LEARNER
I20260927 07:07:02.963518 5288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } }
I20260927 07:07:02.964730 5288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:02.964444 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:02.965050 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: LEARNER
I20260927 07:07:02.974858 5302 catalog_manager.cc:1259] Loaded cluster ID: aedf900fd14c47578e5a439c434c5244
I20260927 07:07:02.974982 5302 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: loading cluster ID for follower catalog manager: success
I20260927 07:07:02.979082 5302 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: acquiring CA information for follower catalog manager: success
I20260927 07:07:02.981668 5302 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:07:03.201920 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.195903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54532)
0927 07:07:03.196206 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:03.196246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:03.196400 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:03.196941 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:03.196946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:03.196976 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:03.197232 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.197247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.198587 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.198592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.199849 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.199863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.201029 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.201035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.201237 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.201279 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 07:07:03.201407 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 07:07:03.201575 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260927 07:07:03.201982 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.196051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.30.16.124:46635)
0927 07:07:03.196596 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:03.196600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:03.196616 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:03.196680 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:03.196683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:03.196749 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:03.196918 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:03.197423 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.198400 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.200014 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.200829 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.200992 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.201237 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:03.201472 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 07:07:03.201620 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":295,"threads_started":1}
I20260927 07:07:03.203303 5263 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 07:07:03.203965 5291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a043bcea9144a32a230edcf5cdb242b. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:03.204252 5291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: LEARNER
W20260927 07:07:03.254262 5236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:03.254537 5236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:03.254729 5236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:03.262490 5236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:03.262676 5236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:03.262861 5236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:03.274264 5236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:41273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=42345
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:03.275701 5236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:03.277124 5236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:03.292354 5308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:03.292502 5311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:03.292333 5309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:03.296237 5236 server_base.cc:1034] running on GCE node
I20260927 07:07:03.296900 5236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:03.297955 5236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:03.299157 5236 hybrid_clock.cc:648] HybridClock initialized: now 1790492823299106 us; error 47 us; skew 500 ppm
I20260927 07:07:03.299512 5236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:03.302865 5236 init.cc:377] Logged in from keytab as oryx/127.30.16.65@KRBTEST.COM (short username oryx)
I20260927 07:07:03.304450 5236 webserver.cc:460] Webserver started at http://127.30.16.65:42345/ using document root <none> and password file <none>
I20260927 07:07:03.305263 5236 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:03.305440 5236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:03.311693 5236 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:07:03.315332 5318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.317031 5236 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 07:07:03.317236 5236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "3f3a6d8b860944a79548ab3b133a9c75"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "500553336eb737c82782600648b80edb"
server_key_iv: "3b754c2dd5db6e1b084eb29ac5529a22"
server_key_version: "encryptionkey@0"
I20260927 07:07:03.318300 5236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:03.333256 5236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:03.337730 5236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:03.338056 5236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:03.339133 5236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:03.340626 5236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:03.340713 5236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.340790 5236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:03.340842 5236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.374907 5236 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41273
I20260927 07:07:03.374984 5430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41273 every 8 connection(s)
I20260927 07:07:03.376401 5236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:03.384439 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5236
I20260927 07:07:03.384851 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4663
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.66@KRBTEST.COM
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
I20260927 07:07:03.423635 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.379401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:37459 (local address 127.30.16.126:39821)
0927 07:07:03.396592 (+ 17191us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:03.396602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:03.397404 (+ 802us) server_negotiation.cc:407] Connection header received
0927 07:07:03.400689 (+ 3285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:03.400697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:03.400811 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:03.400944 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:03.404022 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.405122 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.408009 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.408837 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.408975 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.414657 (+ 5682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:03.414672 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:03.414676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:03.414711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:03.417514 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.419979 (+ 2465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.419987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.419992 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.420144 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.421338 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.421345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.421351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.421726 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:03.421862 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:03.422953 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0927 07:07:03.423219 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17013,"thread_start_us":17887,"threads_started":1}
I20260927 07:07:03.424657 5303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.397440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:59817 (local address 127.30.16.124:46635)
0927 07:07:03.397695 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:03.397701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:03.402019 (+ 4318us) server_negotiation.cc:407] Connection header received
0927 07:07:03.402331 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:03.402337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:03.402438 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:03.402537 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:03.404061 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.404856 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.408049 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.408802 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.408940 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.414715 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:03.414722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:03.414729 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:03.414775 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:03.418445 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.419618 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.419629 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.419636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.419791 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.420726 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.420731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.420734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.421130 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:03.421265 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:03.424070 (+ 2805us) server_negotiation.cc:299] Negotiation successful
0927 07:07:03.424322 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 07:07:03.425226 5437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.380239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:37459)
0927 07:07:03.397146 (+ 16907us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:03.397166 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:03.400250 (+ 3084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:03.401301 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:03.401314 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:03.402639 (+ 1325us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:03.403806 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.403835 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.405294 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.405300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.407811 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.407844 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.409203 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.409209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.409602 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.410586 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:03.410630 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:03.414325 (+ 3695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:03.418673 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.418682 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.418687 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.419506 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.420296 (+ 790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.420302 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.420307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.420851 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.421846 (+ 995us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:03.421856 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:03.422400 (+ 544us) client_negotiation.cc:769] Sending connection context
0927 07:07:03.422955 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 07:07:03.423581 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16742,"mutex_wait_us":290,"spinlock_wait_cycles":59392,"thread_start_us":15004,"threads_started":1}
I20260927 07:07:03.426148 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.397855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.30.16.65:59817)
0927 07:07:03.401890 (+ 4035us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:03.401916 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:03.402145 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:03.402856 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:03.402863 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:03.403463 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:03.403933 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.403948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.405015 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.405021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.407832 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.407844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.409105 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.409114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.410509 (+ 1395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.411234 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:03.411260 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:03.414431 (+ 3171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:03.418626 (+ 4195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.418637 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.418658 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.419471 (+ 813us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.420312 (+ 841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.420317 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.420320 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.420591 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.421939 (+ 1348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:03.421943 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:03.422402 (+ 459us) client_negotiation.cc:769] Sending connection context
0927 07:07:03.424847 (+ 2445us) client_negotiation.cc:240] Negotiation successful
0927 07:07:03.425147 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3879,"mutex_wait_us":253,"spinlock_wait_cycles":47488,"thread_start_us":3719,"threads_started":1}
I20260927 07:07:03.426208 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.379363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:52651 (local address 127.30.16.125:39935)
0927 07:07:03.401553 (+ 22190us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:03.401562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:03.401592 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:07:03.401649 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:03.401654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:03.401734 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:03.401846 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:03.405304 (+ 3458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.406485 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.409951 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.410557 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.410739 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.418415 (+ 7676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:03.418422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:03.418426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:03.418469 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:03.422024 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.422941 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.422946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.422950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.423114 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.423597 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.423602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.423605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.423964 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:03.424102 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:03.425526 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0927 07:07:03.425768 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22032,"thread_start_us":21884,"threads_started":1}
I20260927 07:07:03.425310 5438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.380239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.30.16.65:52651)
0927 07:07:03.394982 (+ 14743us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:03.395065 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:03.400767 (+ 5702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:03.402092 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:03.402096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:03.402640 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:03.403805 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.403832 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.406687 (+ 2855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.406691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.409726 (+ 3035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:03.409740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.411042 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.411051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:03.411284 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.411881 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:03.411904 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:03.418143 (+ 6239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:03.422197 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.422202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.422207 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.422776 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.423254 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:03.423259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:03.423264 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:03.423454 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:03.424176 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:03.424188 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:03.424380 (+ 192us) client_negotiation.cc:769] Sending connection context
0927 07:07:03.424524 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 07:07:03.424839 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14217,"mutex_wait_us":203,"thread_start_us":16388,"threads_started":1}
I20260927 07:07:03.427830 5431 heartbeater.cc:347] Connected to a master server at 127.30.16.124:46635
I20260927 07:07:03.427871 5432 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:03.427901 5433 heartbeater.cc:347] Connected to a master server at 127.30.16.125:39935
I20260927 07:07:03.428197 5431 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:03.428228 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:03.386819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.30.16.66:43105)
0927 07:07:03.398037 (+ 11218us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:03.398049 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:03.399014 (+ 965us) server_negotiation.cc:407] Connection header received
0927 07:07:03.399122 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:03.399134 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:03.399478 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:03.399682 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:03.403819 (+ 4137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.405376 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.407265 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:03.409290 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:03.409421 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:03.417580 (+ 8159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:03.417602 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:03.417625 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:03.417681 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:03.421661 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.422728 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.422738 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.422744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.422982 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:03.423412 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:03.423418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:03.423422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:03.424125 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:03.424408 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:03.427542 (+ 3134us) server_negotiation.cc:299] Negotiation successful
0927 07:07:03.427825 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11016,"thread_start_us":10848,"threads_started":1}
I20260927 07:07:03.428215 5432 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:03.428218 5433 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:03.429241 5431 heartbeater.cc:511] Master 127.30.16.124:46635 requested a full tablet report, sending...
I20260927 07:07:03.429239 5432 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
I20260927 07:07:03.429883 4786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:03.429937 5433 heartbeater.cc:511] Master 127.30.16.125:39935 requested a full tablet report, sending...
I20260927 07:07:03.431376 4468 ts_manager.cc:194] Re-registered known tserver with Master: 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273)
I20260927 07:07:03.431993 5066 ts_manager.cc:194] Registered new tserver with Master: 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273)
I20260927 07:07:03.432017 5253 ts_manager.cc:194] Registered new tserver with Master: 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273)
I20260927 07:07:03.432483 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:37459
I20260927 07:07:03.517529 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4663
I20260927 07:07:03.534852 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:43105
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:39821
--webserver_port=35535
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:03.740402 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: NON_VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: true } } }
I20260927 07:07:03.740893 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:03.741303 5446 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: attempting to promote NON_VOTER 49632a44d46b4a7f8062296b9ba3ada5 to VOTER
I20260927 07:07:03.742273 5446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } }
I20260927 07:07:03.744771 5263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 LEARNER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 07:07:03.744782 5076 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Refusing update from remote peer 5a043bcea9144a32a230edcf5cdb242b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 07:07:03.745424 5444 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 07:07:03.745849 5447 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [LEADER]: Connected to new peer: Peer: permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 07:07:03.748636 5447 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.749784 5076 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.752449 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.752753 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:03.753096 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.753355 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:03.760232 5451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.760803 5451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:03.761174 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.761467 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:03.761967 5263 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.765579 5452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a043bcea9144a32a230edcf5cdb242b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:03.766000 5452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:07:03.874951 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:03.875241 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:03.875414 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:03.882936 5443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:03.883059 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:03.883248 5443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:03.893299 5443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:43105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=35535
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:03.894765 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:03.896206 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:03.909580 5457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:03.909691 5458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:03.910140 5460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:03.910609 5443 server_base.cc:1034] running on GCE node
I20260927 07:07:03.911276 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:03.912292 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:03.913517 5443 hybrid_clock.cc:648] HybridClock initialized: now 1790492823913444 us; error 67 us; skew 500 ppm
I20260927 07:07:03.913977 5443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:03 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:03.917548 5443 init.cc:377] Logged in from keytab as oryx/127.30.16.66@KRBTEST.COM (short username oryx)
I20260927 07:07:03.919317 5443 webserver.cc:460] Webserver started at http://127.30.16.66:35535/ using document root <none> and password file <none>
I20260927 07:07:03.920070 5443 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:03.920244 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:03.926836 5443 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 07:07:03.929798 5467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.931406 5443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:03.931593 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "c28b93149fc84515a5645a9697ef73bb"
format_stamp: "Formatted at 2026-09-27 07:07:00 on dist-test-slave-8vqf"
server_key: "37cbdf2c814168c6d99185159002bcec"
server_key_iv: "18fc21c48d1763ddab0d8b3ecfd22fd0"
server_key_version: "encryptionkey@0"
I20260927 07:07:03.932566 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:03.952708 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:03.957358 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:03.957647 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:03.958734 5443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:03.960135 5443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:03.960261 5443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.960408 5443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:03.960498 5443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:03.995844 5443 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:43105
I20260927 07:07:03.995891 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:43105 every 8 connection(s)
I20260927 07:07:03.997362 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:04.006011 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5443
I20260927 07:07:04.006330 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4796
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:04.034119 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.001480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52193 (local address 127.30.16.126:39821)
0927 07:07:04.010794 (+ 9314us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.010804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.010829 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:07:04.010906 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.010914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.011025 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.013916 (+ 2891us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.014152 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.015539 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.018869 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.019722 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.019886 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.026128 (+ 6242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.026143 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.026148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.026192 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.029845 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.030937 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.030943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.030946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.031108 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.031652 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.031657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.031660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.032033 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.032149 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.033445 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.033678 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9146,"thread_start_us":9098,"threads_started":1}
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
I20260927 07:07:04.036734 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.001131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:52193)
0927 07:07:04.005171 (+ 4040us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.005228 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.008976 (+ 3748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.011565 (+ 2589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.011583 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.012602 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.013660 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.013691 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.015798 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.015808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.018598 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.018620 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.020287 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.020297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.020706 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.021829 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.021878 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.025879 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.030032 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.030042 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.030063 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.030767 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.031245 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.031250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.031254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.031521 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.032250 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.032258 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.032756 (+ 498us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.034535 (+ 1779us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.035169 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3712,"mutex_wait_us":6,"thread_start_us":3399,"threads_started":1}
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.67@KRBTEST.COM
I20260927 07:07:04.038717 5581 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:04.039077 5581 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.040074 5581 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492823, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
I20260927 07:07:04.041934 4468 ts_manager.cc:194] Re-registered known tserver with Master: c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105)
I20260927 07:07:04.043196 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:52193
I20260927 07:07:04.047000 5589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.007458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50922 (local address 127.30.16.67:35957)
0927 07:07:04.024467 (+ 17009us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.024481 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.025484 (+ 1003us) server_negotiation.cc:407] Connection header received
0927 07:07:04.025696 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.025706 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.026058 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.026321 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.028971 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.030624 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.032534 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.033115 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.033262 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.039330 (+ 6068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.039343 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.039356 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.039389 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.043770 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.044590 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.044598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.044600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.044735 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.045148 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.045152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.045154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.045538 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.045848 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.046207 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.046486 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16784,"thread_start_us":16607,"threads_started":1}
I20260927 07:07:04.048015 5590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.010386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.30.16.66:33963)
0927 07:07:04.025357 (+ 14971us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.025375 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.025517 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.026792 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.026799 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.027424 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.027732 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.027743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.029328 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.029333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.030921 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.030932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.032133 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.032139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.033098 (+ 959us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.037853 (+ 4755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.037878 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.041342 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.045465 (+ 4123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.045470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.045473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.045977 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.046433 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.046438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.046442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.046621 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.047293 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.047297 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.047421 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.047530 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.047728 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14845,"mutex_wait_us":476,"thread_start_us":14699,"threads_started":1}
I20260927 07:07:04.048417 5591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.010509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:33963 (local address 127.30.16.125:39935)
0927 07:07:04.024906 (+ 14397us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.024912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.025631 (+ 719us) server_negotiation.cc:407] Connection header received
0927 07:07:04.025703 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.025708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.025803 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.025904 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.027918 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.029131 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.031087 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.031762 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.031877 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.041579 (+ 9702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.041586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.041591 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.041651 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.045243 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.046159 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.046165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.046168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.046307 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.046749 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.046754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.046757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.047114 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.047297 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.047687 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.047911 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14289,"thread_start_us":14117,"threads_started":1}
I20260927 07:07:04.048727 4919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:04.049552 5582 heartbeater.cc:347] Connected to a master server at 127.30.16.125:39935
I20260927 07:07:04.049732 5582 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.050187 5582 heartbeater.cc:511] Master 127.30.16.125:39935 requested a full tablet report, sending...
I20260927 07:07:04.051398 5066 ts_manager.cc:194] Registered new tserver with Master: c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105)
I20260927 07:07:04.055631 5584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.000310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:40217 (local address 127.30.16.124:46635)
0927 07:07:04.027675 (+ 27365us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.027683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.027703 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:07:04.027763 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.027768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.027863 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.029259 (+ 1396us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.029439 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.030585 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.032645 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.033307 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.033959 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.037700 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.037710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.037715 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.037757 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.043883 (+ 6126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.053474 (+ 9591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.053483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.053489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.053657 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.054088 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.054094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.054098 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.054395 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.054554 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.054982 (+ 428us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.055214 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27220,"thread_start_us":27695,"threads_started":1}
I20260927 07:07:04.055644 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.001131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.30.16.66:40217)
0927 07:07:04.010139 (+ 9008us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.010166 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.010301 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.028207 (+ 17906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.028214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.028748 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.029099 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.029113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.030776 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.030782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.032404 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.032415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.033506 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.033513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.033683 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.034518 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.034541 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.037158 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.044054 (+ 6896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.044060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.044065 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.044624 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.053799 (+ 9175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.053804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.053809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.053964 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.054532 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.054536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.054727 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.054878 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.055213 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8777,"mutex_wait_us":521,"thread_start_us":8061,"threads_started":1}
I20260927 07:07:04.056497 5580 heartbeater.cc:347] Connected to a master server at 127.30.16.124:46635
I20260927 07:07:04.056655 5580 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.057061 5580 heartbeater.cc:511] Master 127.30.16.124:46635 requested a full tablet report, sending...
I20260927 07:07:04.058120 5253 ts_manager.cc:194] Registered new tserver with Master: c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105)
I20260927 07:07:04.134590 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4796
I20260927 07:07:04.156082 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:35957
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:39821
--webserver_port=41961
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:46039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:04.435631 5432 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
W20260927 07:07:04.502421 5593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:04.502763 5593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:04.502933 5593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:04.510315 5593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:04.510443 5593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:04.510666 5593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:04.521229 5593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:46039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:35957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=41961
--tserver_master_addrs=127.30.16.126:39821,127.30.16.125:39935,127.30.16.124:46635
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:04.522828 5593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:04.524230 5593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:04.538884 5600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.539449 5602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:04.539573 5599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:04.540520 5593 server_base.cc:1034] running on GCE node
I20260927 07:07:04.541242 5593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:04.542580 5593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:04.543808 5593 hybrid_clock.cc:648] HybridClock initialized: now 1790492824543749 us; error 47 us; skew 500 ppm
I20260927 07:07:04.544267 5593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:04.548238 5593 init.cc:377] Logged in from keytab as oryx/127.30.16.67@KRBTEST.COM (short username oryx)
I20260927 07:07:04.550040 5593 webserver.cc:460] Webserver started at http://127.30.16.67:41961/ using document root <none> and password file <none>
I20260927 07:07:04.550805 5593 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:04.550966 5593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:04.556425 5593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 07:07:04.559269 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.560745 5593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:04.560930 5593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "f96b510a824d44899a323c0956d721cd"
format_stamp: "Formatted at 2026-09-27 07:07:01 on dist-test-slave-8vqf"
server_key: "a1c0de4b80c91a5129bdc37f22f70037"
server_key_iv: "0cf384853a40f642d6c5a056f61408f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:04.561905 5593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:04.586113 5593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:04.590839 5593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:04.591140 5593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:04.592162 5593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:04.593678 5593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:04.593796 5593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.593931 5593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:04.594026 5593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:04.628402 5593 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:35957
I20260927 07:07:04.628495 5721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:35957 every 8 connection(s)
I20260927 07:07:04.629891 5593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:04.630420 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5593
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.124@KRBTEST.COM
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.125@KRBTEST.COM
Sep 27 07:07:04 dist-test-slave-8vqf krb5kdc[4423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492824, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:04.693154 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.633068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:53843 (local address 127.30.16.126:39821)
0927 07:07:04.633831 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.633835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.646455 (+ 12620us) server_negotiation.cc:407] Connection header received
0927 07:07:04.660287 (+ 13832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.660296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.660412 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.660522 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.664541 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.666082 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.676885 (+ 10803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.677536 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.677670 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.684580 (+ 6910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.684587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.684593 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.684653 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.688975 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.690055 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.690062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.690066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.690215 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.690896 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.690904 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.690907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.691270 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.691416 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.692503 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.692740 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":649,"thread_start_us":556,"threads_started":1}
I20260927 07:07:04.694518 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.634005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:53843)
0927 07:07:04.645491 (+ 11486us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.646245 (+ 754us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.660084 (+ 13839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.660544 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.660548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.663804 (+ 3256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.664293 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.664315 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.666509 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.666517 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.676647 (+ 10130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.676666 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.678448 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.678456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.678751 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.681602 (+ 2851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.681628 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.684357 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.689112 (+ 4755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.689124 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.689146 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.689835 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.690325 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.690330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.690334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.690674 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.691408 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.691419 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.691976 (+ 557us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.692317 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.692958 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11025,"mutex_wait_us":696,"spinlock_wait_cycles":140160,"thread_start_us":11653,"threads_started":1}
I20260927 07:07:04.696257 5723 heartbeater.cc:347] Connected to a master server at 127.30.16.126:39821
I20260927 07:07:04.696372 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.634491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.30.16.67:45015)
0927 07:07:04.646127 (+ 11636us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.646161 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.655876 (+ 9715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.659360 (+ 3484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.659380 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.660773 (+ 1393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.661970 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.662001 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.663754 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.663759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.667289 (+ 3530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.667305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.670264 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.670275 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.671017 (+ 742us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.672048 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.672088 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.680281 (+ 8193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.692553 (+ 12272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.692557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.692561 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.693061 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.694336 (+ 1275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.694340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.694343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.694538 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.695309 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.695313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.695474 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.695658 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.695940 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11523,"mutex_wait_us":9,"thread_start_us":10409,"threads_started":1}
I20260927 07:07:04.696612 5723 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.697192 5732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.647070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:39921 (local address 127.30.16.125:39935)
0927 07:07:04.662858 (+ 15788us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.662867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.665195 (+ 2328us) server_negotiation.cc:407] Connection header received
0927 07:07:04.667491 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.667500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.667616 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.667699 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.669798 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.671561 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.674474 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.675238 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.675383 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.681275 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.681284 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.681291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.681345 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.689766 (+ 8421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.690577 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.690582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.690587 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.690755 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.691836 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.691841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.691845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.692196 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.692332 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.696212 (+ 3880us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.696818 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15603,"thread_start_us":15794,"threads_started":1}
I20260927 07:07:04.697332 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.634499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45015 (local address 127.30.16.124:46635)
0927 07:07:04.658375 (+ 23876us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.658386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.658412 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:07:04.658494 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.658502 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.658651 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.662175 (+ 3524us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.662369 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.663575 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.667893 (+ 4318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.669876 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.675896 (+ 6020us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.680599 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.680607 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.680612 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.680659 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.692243 (+ 11584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.694039 (+ 1796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.694044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.694047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.694194 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.694704 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.694708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.694711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.695177 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.696561 (+ 1384us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.696724 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.696985 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23695,"thread_start_us":23524,"threads_started":1}
I20260927 07:07:04.697551 5723 heartbeater.cc:511] Master 127.30.16.126:39821 requested a full tablet report, sending...
I20260927 07:07:04.698216 5722 heartbeater.cc:347] Connected to a master server at 127.30.16.124:46635
I20260927 07:07:04.698359 5722 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.698817 5722 heartbeater.cc:511] Master 127.30.16.124:46635 requested a full tablet report, sending...
I20260927 07:07:04.699355 4468 ts_manager.cc:194] Re-registered known tserver with Master: f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:04.699996 5253 ts_manager.cc:194] Registered new tserver with Master: f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:04.700564 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.647160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.30.16.67:39921)
0927 07:07:04.663378 (+ 16218us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.663403 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.667276 (+ 3873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.668063 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.668071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.668879 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:04.669428 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.669445 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.671784 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.671792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.674281 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.674301 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.675719 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.675727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.675974 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.677780 (+ 1806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:04.677808 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:04.680997 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:04.689910 (+ 8913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.689913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.689916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.690441 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.691499 (+ 1058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:04.691506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:04.691510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:04.691692 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:04.693773 (+ 2081us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:04.693780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:04.693941 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.697810 (+ 3869us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.698108 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16094,"mutex_wait_us":3,"thread_start_us":15966,"threads_started":1}
I20260927 07:07:04.700587 4468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:53843
I20260927 07:07:04.701656 5724 heartbeater.cc:347] Connected to a master server at 127.30.16.125:39935
I20260927 07:07:04.701788 5724 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:04.702322 5724 heartbeater.cc:511] Master 127.30.16.125:39935 requested a full tablet report, sending...
I20260927 07:07:04.703459 5066 ts_manager.cc:194] Registered new tserver with Master: f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:04.704646 5738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.667298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.30.16.125:39935)
0927 07:07:04.677947 (+ 10649us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.677953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.677974 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:07:04.678030 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.678035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.678104 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.682613 (+ 4509us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.682876 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.685161 (+ 2285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.687402 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.688800 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.689472 (+ 672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.693264 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.693271 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.693279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.693315 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.700103 (+ 6788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.701567 (+ 1464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.701573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.701579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.701780 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.702267 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.702271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.702274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.702694 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.703865 (+ 1171us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.703980 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.704206 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10498,"thread_start_us":10976,"threads_started":1}
I20260927 07:07:04.710367 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.663708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.30.16.126:39821)
0927 07:07:04.673830 (+ 10122us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.673836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.673856 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:07:04.673915 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.673922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.674010 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.674115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.676456 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.679537 (+ 3081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.692398 (+ 12861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.693094 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.693232 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.703891 (+ 10659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.703898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.703903 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.703939 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.707586 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.708290 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.708298 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.708302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.708479 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.708826 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.708832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.708839 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.709195 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.709457 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.709696 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.709941 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1791,"server-negotiator.queue_time_us":9977,"thread_start_us":9888,"threads_started":1}
I20260927 07:07:04.714373 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.677931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54540 (local address 127.30.16.124:46635)
0927 07:07:04.691086 (+ 13155us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.691093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.694037 (+ 2944us) server_negotiation.cc:407] Connection header received
0927 07:07:04.694099 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.694106 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.694190 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.694336 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:04.696580 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.701440 (+ 4860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.703342 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.703992 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.704166 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.708183 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:04.708189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:04.708192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:04.708251 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:04.711613 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.712346 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.712356 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.712361 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.712523 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:04.713047 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:04.713054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:04.713058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:04.713341 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:04.713518 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.713928 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.714129 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":11001,"threads_started":1}
I20260927 07:07:04.714838 4468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:04.723322 4468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:04.778297 5752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.761911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:35957 (local address 127.0.0.1:50928)
0927 07:07:04.762506 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.762562 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.762670 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.764270 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.764278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.764313 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.764761 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.764777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.767920 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.767925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.775042 (+ 7117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.775063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.777188 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.777194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.777431 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.777483 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.777589 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.777783 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":281,"threads_started":1}
I20260927 07:07:04.779480 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.762585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50928 (local address 127.30.16.67:35957)
0927 07:07:04.763449 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.763460 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.763495 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:07:04.763616 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.763628 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.764009 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.765042 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.765381 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.767735 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.775260 (+ 7525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.776973 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.777126 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.777375 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.778802 (+ 1427us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.779018 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":485,"threads_started":1}
I20260927 07:07:04.779938 5751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.760205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:43105 (local address 127.0.0.1:55654)
0927 07:07:04.761230 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.761256 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.761414 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.772213 (+ 10799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.772221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.772257 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.772764 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.772780 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.775137 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.775143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.777197 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.777211 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.779141 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.779150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.779332 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.779378 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.779444 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.779568 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":359,"threads_started":1}
I20260927 07:07:04.780941 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.760286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.30.16.66:43105)
0927 07:07:04.771306 (+ 11020us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.771315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.771344 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:07:04.771456 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.771475 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.771764 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.773019 (+ 1255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.773256 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.774934 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.777398 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.778905 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.780202 (+ 1297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.780420 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.780496 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.780667 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10856,"thread_start_us":11649,"threads_started":1}
I20260927 07:07:04.784251 5515 tablet_service.cc:1467] Processing CreateTablet for tablet d7614cec6bb34fb992c3f617906e65bc (DEFAULT_TABLE table=test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.784097 5657 tablet_service.cc:1467] Processing CreateTablet for tablet d7614cec6bb34fb992c3f617906e65bc (DEFAULT_TABLE table=test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.785081 5753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.762543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41273 (local address 127.0.0.1:49816)
0927 07:07:04.773596 (+ 11053us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:04.773616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:04.773738 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:04.774488 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:04.774495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:04.774519 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:04.775127 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.775138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.778119 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.778125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.781400 (+ 3275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:04.781413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.784296 (+ 2883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.784303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:04.784508 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.784539 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:07:04.784618 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:07:04.784751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10941,"thread_start_us":10724,"threads_started":1}
I20260927 07:07:04.785681 5515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7614cec6bb34fb992c3f617906e65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.785693 5657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7614cec6bb34fb992c3f617906e65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.785802 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:04.763069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.30.16.65:41273)
0927 07:07:04.771072 (+ 8003us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:04.771085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:04.773739 (+ 2654us) server_negotiation.cc:407] Connection header received
0927 07:07:04.773927 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:04.773942 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:04.774245 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:04.774493 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:04.775352 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.776885 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.781571 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:04.783828 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:04.783959 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:04.784152 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:04.785320 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 07:07:04.785509 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7821,"thread_start_us":7709,"threads_started":1}
I20260927 07:07:04.790275 5366 tablet_service.cc:1467] Processing CreateTablet for tablet d7614cec6bb34fb992c3f617906e65bc (DEFAULT_TABLE table=test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:04.791709 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7614cec6bb34fb992c3f617906e65bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:04.801668 5756 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Bootstrap starting.
I20260927 07:07:04.803280 5757 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Bootstrap starting.
I20260927 07:07:04.805246 5756 tablet_bootstrap.cc:654] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.806515 5756 log.cc:824] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.806705 5757 tablet_bootstrap.cc:654] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.808110 5757 log.cc:824] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.811470 5758 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Bootstrap starting.
I20260927 07:07:04.811913 5757 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: No bootstrap required, opened a new log
I20260927 07:07:04.812206 5757 ts_tablet_manager.cc:1397] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260927 07:07:04.814465 5756 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: No bootstrap required, opened a new log
I20260927 07:07:04.814859 5756 ts_tablet_manager.cc:1397] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 07:07:04.814997 5758 tablet_bootstrap.cc:654] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:04.816006 5758 log.cc:824] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:04.819097 5758 tablet_bootstrap.cc:492] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: No bootstrap required, opened a new log
I20260927 07:07:04.819387 5758 ts_tablet_manager.cc:1397] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 07:07:04.823031 5757 raft_consensus.cc:348] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.823520 5757 raft_consensus.cc:374] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.823625 5757 raft_consensus.cc:729] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28b93149fc84515a5645a9697ef73bb, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.824290 5757 consensus_queue.cc:260] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.825778 5756 raft_consensus.cc:348] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.826211 5757 ts_tablet_manager.cc:1428] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 07:07:04.826238 5756 raft_consensus.cc:374] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.826908 5581 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
I20260927 07:07:04.826326 5756 raft_consensus.cc:729] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96b510a824d44899a323c0956d721cd, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.828176 5756 consensus_queue.cc:260] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.829282 5758 raft_consensus.cc:348] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.829739 5758 raft_consensus.cc:374] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:04.829833 5758 raft_consensus.cc:729] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3a6d8b860944a79548ab3b133a9c75, State: Initialized, Role: FOLLOWER
I20260927 07:07:04.830443 5758 consensus_queue.cc:260] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:04.837060 5756 ts_tablet_manager.cc:1428] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Time spent starting tablet: real 0.022s user 0.009s sys 0.009s
I20260927 07:07:04.837110 5758 ts_tablet_manager.cc:1428] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260927 07:07:04.840277 5723 heartbeater.cc:503] Master 127.30.16.126:39821 was elected leader, sending a full tablet report...
W20260927 07:07:04.880296 5434 tablet.cc:2369] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:04.883798 5726 tablet.cc:2369] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:05.008611 5583 tablet.cc:2369] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:05.202533 5764 raft_consensus.cc:482] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:05.202904 5764 raft_consensus.cc:504] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:05.204795 5764 leader_election.cc:288] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105), f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:05.212603 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.205219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:55027 (local address 127.30.16.67:35957)
0927 07:07:05.205450 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.205459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.206333 (+ 874us) server_negotiation.cc:407] Connection header received
0927 07:07:05.206494 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.206501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.206620 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.206724 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.207454 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.208140 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.210085 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.211353 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.211518 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.211607 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.212212 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.212348 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:07:05.212637 5750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.205505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:49697 (local address 127.30.16.66:43105)
0927 07:07:05.205775 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.205784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.206393 (+ 609us) server_negotiation.cc:407] Connection header received
0927 07:07:05.206542 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.206596 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.206672 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.206770 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:05.207412 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.208125 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.210158 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.211449 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.211597 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.211658 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.212212 (+ 554us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.212360 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:07:05.212819 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.205087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:35957 (local address 127.30.16.65:55027)
0927 07:07:05.206183 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.206217 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.206341 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.206740 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.206745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.206767 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.207295 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.207309 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.208275 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.208281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.209920 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.209931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.211629 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.211638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.211926 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.212069 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.212173 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.212338 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"mutex_wait_us":155,"thread_start_us":887,"threads_started":1}
I20260927 07:07:05.212930 5765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.205131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:43105 (local address 127.30.16.65:49697)
0927 07:07:05.206210 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:05.206224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:05.206387 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:05.206784 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:05.206788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:05.206811 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:05.207226 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.207239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.208268 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.208273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.209918 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:05.209930 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.211661 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.211667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:05.211930 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.212069 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 07:07:05.212173 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:07:05.212369 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":192,"thread_start_us":643,"threads_started":1}
I20260927 07:07:05.213965 5677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7614cec6bb34fb992c3f617906e65bc" candidate_uuid: "3f3a6d8b860944a79548ab3b133a9c75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96b510a824d44899a323c0956d721cd" is_pre_election: true
I20260927 07:07:05.214282 5535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7614cec6bb34fb992c3f617906e65bc" candidate_uuid: "3f3a6d8b860944a79548ab3b133a9c75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28b93149fc84515a5645a9697ef73bb" is_pre_election: true
I20260927 07:07:05.214458 5677 raft_consensus.cc:2454] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3a6d8b860944a79548ab3b133a9c75 in term 0.
I20260927 07:07:05.214829 5535 raft_consensus.cc:2454] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f3a6d8b860944a79548ab3b133a9c75 in term 0.
I20260927 07:07:05.215260 5322 leader_election.cc:302] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3a6d8b860944a79548ab3b133a9c75, f96b510a824d44899a323c0956d721cd; no voters:
I20260927 07:07:05.215731 5764 raft_consensus.cc:2781] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:05.215871 5764 raft_consensus.cc:482] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:05.215986 5764 raft_consensus.cc:3037] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.218173 5764 raft_consensus.cc:504] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:05.218911 5764 leader_election.cc:288] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [CANDIDATE]: Term 1 election: Requested vote from peers c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105), f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:05.219364 5535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7614cec6bb34fb992c3f617906e65bc" candidate_uuid: "3f3a6d8b860944a79548ab3b133a9c75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c28b93149fc84515a5645a9697ef73bb"
I20260927 07:07:05.219450 5677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7614cec6bb34fb992c3f617906e65bc" candidate_uuid: "3f3a6d8b860944a79548ab3b133a9c75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96b510a824d44899a323c0956d721cd"
I20260927 07:07:05.219583 5535 raft_consensus.cc:3037] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.219696 5677 raft_consensus.cc:3037] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:05.221956 5677 raft_consensus.cc:2454] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3a6d8b860944a79548ab3b133a9c75 in term 1.
I20260927 07:07:05.221964 5535 raft_consensus.cc:2454] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f3a6d8b860944a79548ab3b133a9c75 in term 1.
I20260927 07:07:05.222594 5322 leader_election.cc:302] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3a6d8b860944a79548ab3b133a9c75, f96b510a824d44899a323c0956d721cd; no voters:
I20260927 07:07:05.222950 5764 raft_consensus.cc:2781] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:05.223537 5764 raft_consensus.cc:686] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 LEADER]: Becoming Leader. State: Replica: 3f3a6d8b860944a79548ab3b133a9c75, State: Running, Role: LEADER
I20260927 07:07:05.224086 5764 consensus_queue.cc:237] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } }
I20260927 07:07:05.228592 4468 catalog_manager.cc:5495] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "3f3a6d8b860944a79548ab3b133a9c75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } health_report { overall_health: HEALTHY } } }
I20260927 07:07:05.350379 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.344516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49824 (local address 127.30.16.65:41273)
0927 07:07:05.345151 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.345159 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.345181 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:07:05.345296 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.345300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.345409 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.345618 (+ 209us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:05.346025 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.346794 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.348079 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.348555 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.348685 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.349189 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:05.349786 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:05.349952 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.350003 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.350082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":404,"threads_started":1}
I20260927 07:07:05.358740 5677 raft_consensus.cc:1261] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Refusing update from remote peer 3f3a6d8b860944a79548ab3b133a9c75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:05.358744 5535 raft_consensus.cc:1261] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Refusing update from remote peer 3f3a6d8b860944a79548ab3b133a9c75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:05.359750 5767 consensus_queue.cc:1035] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:05.360111 5764 consensus_queue.cc:1035] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:05.373749 5772 mvcc.cc:205] Tried to move back new op lower bound from 7333858612659105792 to 7333858612121780224. Current Snapshot: MvccSnapshot[applied={T|T < 7333858612659105792}]
I20260927 07:07:05.399118 4466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:05.399421 4466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:05.406724 4466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a043bcea9144a32a230edcf5cdb242b: Sending DeleteTablet for 3 replicas of tablet d7614cec6bb34fb992c3f617906e65bc
I20260927 07:07:05.408643 5515 tablet_service.cc:1514] Processing DeleteTablet for tablet d7614cec6bb34fb992c3f617906e65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55654
I20260927 07:07:05.409345 5657 tablet_service.cc:1514] Processing DeleteTablet for tablet d7614cec6bb34fb992c3f617906e65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:50928
I20260927 07:07:05.409559 5366 tablet_service.cc:1514] Processing DeleteTablet for tablet d7614cec6bb34fb992c3f617906e65bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:05 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:49816
I20260927 07:07:05.423098 5783 tablet_replica.cc:330] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: stopping tablet replica
I20260927 07:07:05.423666 5783 raft_consensus.cc:2229] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:05.424248 5783 raft_consensus.cc:2258] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.428505 5783 ts_tablet_manager.cc:1905] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.439603 5785 tablet_replica.cc:330] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: stopping tablet replica
I20260927 07:07:05.440393 5785 raft_consensus.cc:2229] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:05.441319 5783 ts_tablet_manager.cc:1918] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.441504 5783 log.cc:1196] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/d7614cec6bb34fb992c3f617906e65bc
I20260927 07:07:05.442088 5783 ts_tablet_manager.cc:1939] T d7614cec6bb34fb992c3f617906e65bc P c28b93149fc84515a5645a9697ef73bb: Deleting consensus metadata
I20260927 07:07:05.442974 5785 raft_consensus.cc:2258] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.444154 4454 catalog_manager.cc:4841] TS c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105): tablet d7614cec6bb34fb992c3f617906e65bc (table test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]) successfully deleted
I20260927 07:07:05.445755 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4436
I20260927 07:07:05.447026 4498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:05.453644 5785 ts_tablet_manager.cc:1905] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.463202 5784 tablet_replica.cc:330] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: stopping tablet replica
I20260927 07:07:05.463768 5784 raft_consensus.cc:2229] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:05.464350 5784 raft_consensus.cc:2258] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:05.466969 5784 ts_tablet_manager.cc:1905] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:05.471145 5785 ts_tablet_manager.cc:1918] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.471344 5785 log.cc:1196] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/d7614cec6bb34fb992c3f617906e65bc
I20260927 07:07:05.472015 5785 ts_tablet_manager.cc:1939] T d7614cec6bb34fb992c3f617906e65bc P 3f3a6d8b860944a79548ab3b133a9c75: Deleting consensus metadata
I20260927 07:07:05.480964 5784 ts_tablet_manager.cc:1918] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:05.481195 5784 log.cc:1196] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/d7614cec6bb34fb992c3f617906e65bc
I20260927 07:07:05.481771 5784 ts_tablet_manager.cc:1939] T d7614cec6bb34fb992c3f617906e65bc P f96b510a824d44899a323c0956d721cd: Deleting consensus metadata
I20260927 07:07:05.482228 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.412094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.30.16.125:39935)
0927 07:07:05.437120 (+ 25026us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.437131 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.438401 (+ 1270us) server_negotiation.cc:407] Connection header received
0927 07:07:05.438498 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.438507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.438657 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.438785 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.456792 (+ 18007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.458402 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.467823 (+ 9421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.468566 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.468733 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.473298 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.473307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.473313 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.473360 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.477903 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.478891 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.478897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.478901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.479071 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.479597 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.479602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.479605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.479979 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.480194 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.481588 (+ 1394us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.481835 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24796,"thread_start_us":25056,"threads_started":1}
I20260927 07:07:05.497584 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.445316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.30.16.124:46635)
0927 07:07:05.463289 (+ 17973us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.463298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.483928 (+ 20630us) server_negotiation.cc:407] Connection header received
0927 07:07:05.484103 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.484107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.484191 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.484312 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.485805 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.486719 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.488097 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.488689 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.488857 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.491884 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.491898 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.491902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.491937 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.494748 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.495402 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.495409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.495412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.495550 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.495998 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.496003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.496006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.496281 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.496560 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.496856 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.497091 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17826,"thread_start_us":19243,"threads_started":1}
I20260927 07:07:05.624939 4455 catalog_manager.cc:4841] TS f96b510a824d44899a323c0956d721cd (127.30.16.67:35957): tablet d7614cec6bb34fb992c3f617906e65bc (table test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]) successfully deleted
I20260927 07:07:05.625283 4455 catalog_manager.cc:4841] TS 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273): tablet d7614cec6bb34fb992c3f617906e65bc (table test-table [id=d588679d4f424c2db0f7f71db9cf9fe1]) successfully deleted
I20260927 07:07:05.625838 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4436
I20260927 07:07:05.679244 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.661079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33304 (local address 127.30.16.125:39935)
0927 07:07:05.661343 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.661352 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.662433 (+ 1081us) server_negotiation.cc:407] Connection header received
0927 07:07:05.662959 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.662966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.663048 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.663121 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.665691 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.666740 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.668555 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.669256 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.669406 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.672457 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.672464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.672471 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.672520 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.676075 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.676932 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.676946 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.676950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.677094 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.677548 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.677553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.677557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.677922 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.678165 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.678633 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.678856 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:07:05.679833 5788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:05.662668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.30.16.124:46635)
0927 07:07:05.662856 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:05.662862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:05.663119 (+ 257us) server_negotiation.cc:407] Connection header received
0927 07:07:05.663294 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:05.663300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:05.663402 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:05.663531 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:05.665248 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.666324 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.668628 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:05.669312 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:05.669426 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:05.673023 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:05.673030 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:05.673034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:05.673081 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:05.676339 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.677200 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.677207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.677212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.677358 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:05.677774 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:05.677779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:05.677782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:05.678150 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:05.678374 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:05.678886 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 07:07:05.679103 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 07:07:06.394851 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.393964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:56119)
0927 07:07:06.394459 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.394651 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":270,"threads_started":1}
W20260927 07:07:06.395328 5432 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:39821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.395628 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.394824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:39443)
0927 07:07:06.395316 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.395438 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":632,"threads_started":1}
W20260927 07:07:06.396034 5581 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:39821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.396112 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.395052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:55489)
0927 07:07:06.395790 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.395921 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":234,"threads_started":1}
W20260927 07:07:06.396436 5723 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:39821 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.912612 5800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a043bcea9144a32a230edcf5cdb242b)
I20260927 07:07:06.913131 5800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } }
I20260927 07:07:06.915558 5800 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821), 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124:46635)
I20260927 07:07:06.918200 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.915744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46176)
0927 07:07:06.917548 (+ 1804us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.917798 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1646,"thread_start_us":1487,"threads_started":1}
I20260927 07:07:06.918435 5799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a043bcea9144a32a230edcf5cdb242b)
W20260927 07:07:06.918676 5052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.918985 5799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } }
I20260927 07:07:06.944216 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.916144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:46635 (local address 127.0.0.1:54572)
0927 07:07:06.923083 (+ 6939us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.923105 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:06.923242 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:06.936078 (+ 12836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:06.936084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:06.936196 (+ 112us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:06.936495 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:06.936507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.938128 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.938134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:06.940265 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:06.940279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.943072 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.943080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:06.943370 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:06.943433 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 07:07:06.943517 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 07:07:06.943724 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6641,"thread_start_us":18870,"threads_started":1}
I20260927 07:07:06.945537 5804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.919124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.30.16.124:46635)
0927 07:07:06.935665 (+ 16541us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:06.935672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:06.935698 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:07:06.935776 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:06.935782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:06.935862 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:06.936645 (+ 783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:06.936805 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.937955 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.940469 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.941760 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.944642 (+ 2882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:06.944796 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:06.944857 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 07:07:06.945047 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16415,"thread_start_us":22723,"threads_started":1}
I20260927 07:07:06.946197 5799 leader_election.cc:288] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821), 5e7413d90bde44999e00ce9fa6053b41 (127.30.16.125:39935)
I20260927 07:07:06.946244 5263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e7413d90bde44999e00ce9fa6053b41" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49632a44d46b4a7f8062296b9ba3ada5" is_pre_election: true
I20260927 07:07:06.946661 5263 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e7413d90bde44999e00ce9fa6053b41 in term 1.
I20260927 07:07:06.947230 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.921035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46188)
0927 07:07:06.946791 (+ 25756us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.946977 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25630,"threads_started":1}
I20260927 07:07:06.947425 5053 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49632a44d46b4a7f8062296b9ba3ada5, 5e7413d90bde44999e00ce9fa6053b41; no voters:
I20260927 07:07:06.947863 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 07:07:06.948092 5800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a043bcea9144a32a230edcf5cdb242b)
I20260927 07:07:06.948210 5800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 1 FOLLOWER]: Advancing to term 2
W20260927 07:07:06.951309 5239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.954582 5800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } }
I20260927 07:07:06.955654 5800 leader_election.cc:288] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 election: Requested vote from peers 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821), 49632a44d46b4a7f8062296b9ba3ada5 (127.30.16.124:46635)
I20260927 07:07:06.956290 5263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e7413d90bde44999e00ce9fa6053b41" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49632a44d46b4a7f8062296b9ba3ada5"
I20260927 07:07:06.956526 5263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:07:06.956750 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.956342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46202)
0927 07:07:06.956483 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.956581 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 07:07:06.957338 5052 leader_election.cc:334] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821): Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.958303 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.957925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46214)
0927 07:07:06.958060 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.958163 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 07:07:06.958590 5052 leader_election.cc:334] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821): Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.959800 5263 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e7413d90bde44999e00ce9fa6053b41 in term 2.
I20260927 07:07:06.960299 5053 leader_election.cc:302] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49632a44d46b4a7f8062296b9ba3ada5, 5e7413d90bde44999e00ce9fa6053b41; no voters: 5a043bcea9144a32a230edcf5cdb242b
I20260927 07:07:06.960407 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.959736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46228)
0927 07:07:06.959913 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.960119 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 07:07:06.960726 5800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 2 FOLLOWER]: Leader election won for term 2
W20260927 07:07:06.960655 5239 leader_election.cc:334] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821): Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
I20260927 07:07:06.961297 5800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [term 2 LEADER]: Becoming Leader. State: Replica: 5e7413d90bde44999e00ce9fa6053b41, State: Running, Role: LEADER
I20260927 07:07:06.961702 5800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } }
I20260927 07:07:06.963949 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7413d90bde44999e00ce9fa6053b41. Latest consensus state: current_term: 2 leader_uuid: "5e7413d90bde44999e00ce9fa6053b41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:06.964469 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:06.965426 5811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:06.965934 5808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.952585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.30.16.125:39935)
0927 07:07:06.956427 (+ 3842us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:06.956436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:06.956469 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:07:06.956557 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:06.956566 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:06.956674 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:06.957560 (+ 886us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:06.957865 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.959636 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.962956 (+ 3320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.964588 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.964751 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:06.964878 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:06.965344 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 07:07:06.965498 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3648,"thread_start_us":3403,"threads_started":1}
I20260927 07:07:06.966049 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:06.923711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:39935 (local address 127.0.0.1:33312)
0927 07:07:06.947409 (+ 23698us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:06.947427 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:06.947540 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:06.956915 (+ 9375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:06.956924 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:06.956969 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:06.957355 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:06.957371 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.959867 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.959877 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:06.962760 (+ 2883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:06.962782 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:06.964802 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:06.964811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:06.965097 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:06.965156 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 07:07:06.965273 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 07:07:06.965465 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23606,"thread_start_us":24300,"threads_started":1}
I20260927 07:07:06.969997 5811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:06.970083 5076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49632a44d46b4a7f8062296b9ba3ada5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e7413d90bde44999e00ce9fa6053b41" is_pre_election: true
I20260927 07:07:06.970983 5811 catalog_manager.cc:1259] Loaded cluster ID: aedf900fd14c47578e5a439c434c5244
I20260927 07:07:06.971078 5811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:06.971256 5240 leader_election.cc:302] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49632a44d46b4a7f8062296b9ba3ada5; no voters: 5a043bcea9144a32a230edcf5cdb242b, 5e7413d90bde44999e00ce9fa6053b41
I20260927 07:07:06.972045 5799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:07:06.972323 5811 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:06.973091 5811 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Loaded TSK: 0
I20260927 07:07:06.973927 5811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:07:07.001317 5065 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:07.009502 5065 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:07.014618 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.014021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46238)
0927 07:07:07.014278 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.014384 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 07:07:07.014602 5263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [term 2 FOLLOWER]: Refusing update from remote peer 5e7413d90bde44999e00ce9fa6053b41: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 07:07:07.015439 5809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 07:07:07.015834 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.015404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46240)
0927 07:07:07.015568 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.015649 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 07:07:07.016044 5052 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 -> Peer 5a043bcea9144a32a230edcf5cdb242b (127.30.16.126:39821): Couldn't send request to peer 5a043bcea9144a32a230edcf5cdb242b. Status: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 07:07:07.018994 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e7413d90bde44999e00ce9fa6053b41. Latest consensus state: current_term: 2 leader_uuid: "5e7413d90bde44999e00ce9fa6053b41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:07.019306 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:07.021451 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "5e7413d90bde44999e00ce9fa6053b41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:07.021809 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:07.022742 5813 mvcc.cc:205] Tried to move back new op lower bound from 7333858619443806208 to 7333858619239305216. Current Snapshot: MvccSnapshot[applied={T|T < 7333858619443806208}]
I20260927 07:07:07.026269 5799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "5e7413d90bde44999e00ce9fa6053b41" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a043bcea9144a32a230edcf5cdb242b" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39821 } } peers { permanent_uuid: "5e7413d90bde44999e00ce9fa6053b41" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 39935 } attrs { promote: false } } peers { permanent_uuid: "49632a44d46b4a7f8062296b9ba3ada5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 46635 } attrs { promote: false } } }
I20260927 07:07:07.026670 5799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49632a44d46b4a7f8062296b9ba3ada5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:07:07.067010 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.032358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:43105 (local address 127.0.0.1:55656)
0927 07:07:07.032611 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.032631 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.032733 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.055364 (+ 22631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.055373 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.055418 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:07.055736 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.055753 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.059290 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.059300 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.062051 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.062071 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.065879 (+ 3808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.065890 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.066204 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.066263 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.066345 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.066570 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 07:07:07.068791 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.032514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55656 (local address 127.30.16.66:43105)
0927 07:07:07.054798 (+ 22284us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.054808 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.054838 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:07:07.054935 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.054941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.055029 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.057294 (+ 2265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:07.057538 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.059055 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.062301 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.065650 (+ 3349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.067759 (+ 2109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.067921 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.067984 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.068247 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22098,"thread_start_us":22641,"threads_started":1}
I20260927 07:07:07.070010 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.034510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.30.16.67:35957)
0927 07:07:07.058062 (+ 23552us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.058072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.059002 (+ 930us) server_negotiation.cc:407] Connection header received
0927 07:07:07.059142 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.059151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.059255 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.059367 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:07.060777 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.062315 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.065808 (+ 3493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.067844 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.067993 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.068218 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.069288 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.069502 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23375,"thread_start_us":23555,"threads_started":1}
I20260927 07:07:07.070079 5515 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 (DEFAULT_TABLE table=test-table [id=1b1bddc8574c4889855d4e56f93376cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:07.070708 5515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6f4f7a1e0a410798d932a439ddb887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:07.071147 5819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.034066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:35957 (local address 127.0.0.1:50936)
0927 07:07:07.058677 (+ 24611us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.058703 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.058868 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.059589 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.059601 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.059635 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:07.060480 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.060498 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.062564 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.062573 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.065568 (+ 2995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.065589 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.068447 (+ 2858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.068456 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.068793 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.068847 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.068956 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.070593 (+ 1637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24426,"mutex_wait_us":354,"thread_start_us":25658,"threads_started":1}
I20260927 07:07:07.072091 5657 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 (DEFAULT_TABLE table=test-table [id=1b1bddc8574c4889855d4e56f93376cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:07.072546 5657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6f4f7a1e0a410798d932a439ddb887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:07.079538 5802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.032725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41273 (local address 127.0.0.1:49840)
0927 07:07:07.033038 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.033054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.033153 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.072448 (+ 39295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.072453 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.072496 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:07.072906 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.072917 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.074151 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.074159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.076430 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.076448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.078440 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.078449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.078704 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.078760 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.078834 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.079020 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260927 07:07:07.080786 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.033047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.30.16.65:41273)
0927 07:07:07.072044 (+ 38997us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.072048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.072065 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:07:07.072122 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.072126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.072183 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.073066 (+ 883us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:07.073254 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.074009 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.076634 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.078161 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.079929 (+ 1768us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.080077 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.080190 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.080371 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38876,"thread_start_us":38775,"threads_started":1}
I20260927 07:07:07.081764 5366 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 (DEFAULT_TABLE table=test-table [id=1b1bddc8574c4889855d4e56f93376cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:07.082262 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6f4f7a1e0a410798d932a439ddb887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:07.084199 5821 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Bootstrap starting.
I20260927 07:07:07.085320 5822 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Bootstrap starting.
I20260927 07:07:07.086427 5821 tablet_bootstrap.cc:654] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:07.088033 5822 tablet_bootstrap.cc:654] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:07.095120 5825 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Bootstrap starting.
I20260927 07:07:07.095315 5821 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: No bootstrap required, opened a new log
I20260927 07:07:07.095508 5821 ts_tablet_manager.cc:1397] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.006s
I20260927 07:07:07.096446 5821 raft_consensus.cc:348] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.096745 5821 raft_consensus.cc:374] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:07.096880 5821 raft_consensus.cc:729] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c28b93149fc84515a5645a9697ef73bb, State: Initialized, Role: FOLLOWER
I20260927 07:07:07.097354 5825 tablet_bootstrap.cc:654] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:07.097193 5821 consensus_queue.cc:260] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.098220 5821 ts_tablet_manager.cc:1428] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 07:07:07.099833 5822 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: No bootstrap required, opened a new log
I20260927 07:07:07.100081 5822 ts_tablet_manager.cc:1397] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260927 07:07:07.102777 5822 raft_consensus.cc:348] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.103108 5822 raft_consensus.cc:374] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:07.103240 5822 raft_consensus.cc:729] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f96b510a824d44899a323c0956d721cd, State: Initialized, Role: FOLLOWER
I20260927 07:07:07.103574 5822 consensus_queue.cc:260] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.107651 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.099902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:39371)
0927 07:07:07.107068 (+ 7166us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.107296 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6964,"threads_started":1}
I20260927 07:07:07.109499 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.108887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:53649)
0927 07:07:07.109186 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.109298 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 07:07:07.110777 5822 ts_tablet_manager.cc:1428] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260927 07:07:07.111618 5825 tablet_bootstrap.cc:492] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: No bootstrap required, opened a new log
I20260927 07:07:07.111842 5825 ts_tablet_manager.cc:1397] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260927 07:07:07.112815 5825 raft_consensus.cc:348] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.113168 5825 raft_consensus.cc:374] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:07.113329 5825 raft_consensus.cc:729] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3a6d8b860944a79548ab3b133a9c75, State: Initialized, Role: FOLLOWER
I20260927 07:07:07.113629 5825 consensus_queue.cc:260] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.118348 5724 heartbeater.cc:503] Master 127.30.16.125:39935 was elected leader, sending a full tablet report...
I20260927 07:07:07.119746 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.111441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:47249)
0927 07:07:07.119254 (+ 7813us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.119450 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7631,"thread_start_us":7452,"threads_started":1}
I20260927 07:07:07.119874 5825 ts_tablet_manager.cc:1428] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260927 07:07:07.123202 5433 heartbeater.cc:503] Master 127.30.16.125:39935 was elected leader, sending a full tablet report...
I20260927 07:07:07.123508 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.122935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:51439)
0927 07:07:07.123238 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.123334 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 07:07:07.124706 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.120542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:36877)
0927 07:07:07.124241 (+ 3699us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.124399 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3554,"thread_start_us":3393,"threads_started":1}
I20260927 07:07:07.125994 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.125434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:38919)
0927 07:07:07.125656 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.125780 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":319,"threads_started":1}
I20260927 07:07:07.154649 5826 raft_consensus.cc:482] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:07.154983 5826 raft_consensus.cc:504] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.157163 5826 leader_election.cc:288] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273), f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:07.165565 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.157459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:59417 (local address 127.30.16.65:41273)
0927 07:07:07.157682 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.157689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.157712 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:07:07.158038 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.158045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.158126 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.158233 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:07.159177 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.160165 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.163335 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.164524 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.164715 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.164797 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.165107 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.165244 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:07:07.165642 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.157365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41273 (local address 127.30.16.66:59417)
0927 07:07:07.157596 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.157623 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.157806 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.158301 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.158310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.158394 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:07.158996 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.159007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.160319 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.160326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.163150 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.163169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.164729 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.164736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.164929 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.164958 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.165069 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.165223 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":28}
I20260927 07:07:07.166518 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f4f7a1e0a410798d932a439ddb887" candidate_uuid: "c28b93149fc84515a5645a9697ef73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3a6d8b860944a79548ab3b133a9c75" is_pre_election: true
I20260927 07:07:07.167109 5386 raft_consensus.cc:2454] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28b93149fc84515a5645a9697ef73bb in term 0.
I20260927 07:07:07.167376 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.157959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:50961 (local address 127.30.16.67:35957)
0927 07:07:07.158149 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.158156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.159173 (+ 1017us) server_negotiation.cc:407] Connection header received
0927 07:07:07.159474 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.159480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.159564 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.159671 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:07.160295 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.161240 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.164053 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.165723 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.165892 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.166002 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.166919 (+ 917us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.167076 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:07:07.167562 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.157797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:35957 (local address 127.30.16.66:50961)
0927 07:07:07.159070 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.159091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:07.159281 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:07.159699 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:07.159707 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:07.159738 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:07.160123 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.160135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.161416 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.161423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.163870 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:07.163884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.166383 (+ 2499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.166392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:07.166655 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.166708 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 07:07:07.166862 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 07:07:07.167056 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1106,"mutex_wait_us":32,"thread_start_us":921,"threads_started":1}
I20260927 07:07:07.168035 5471 leader_election.cc:302] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f3a6d8b860944a79548ab3b133a9c75, c28b93149fc84515a5645a9697ef73bb; no voters:
I20260927 07:07:07.168519 5826 raft_consensus.cc:2781] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:07.168656 5826 raft_consensus.cc:482] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:07.168762 5826 raft_consensus.cc:3037] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:07.169001 5677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f4f7a1e0a410798d932a439ddb887" candidate_uuid: "c28b93149fc84515a5645a9697ef73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96b510a824d44899a323c0956d721cd" is_pre_election: true
I20260927 07:07:07.169314 5677 raft_consensus.cc:2454] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c28b93149fc84515a5645a9697ef73bb in term 0.
I20260927 07:07:07.172595 5826 raft_consensus.cc:504] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.173466 5826 leader_election.cc:288] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [CANDIDATE]: Term 1 election: Requested vote from peers 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273), f96b510a824d44899a323c0956d721cd (127.30.16.67:35957)
I20260927 07:07:07.173998 5677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f4f7a1e0a410798d932a439ddb887" candidate_uuid: "c28b93149fc84515a5645a9697ef73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f96b510a824d44899a323c0956d721cd"
I20260927 07:07:07.174024 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6f4f7a1e0a410798d932a439ddb887" candidate_uuid: "c28b93149fc84515a5645a9697ef73bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f3a6d8b860944a79548ab3b133a9c75"
I20260927 07:07:07.174232 5677 raft_consensus.cc:3037] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:07.174283 5386 raft_consensus.cc:3037] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:07.176818 5677 raft_consensus.cc:2454] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28b93149fc84515a5645a9697ef73bb in term 1.
I20260927 07:07:07.177284 5471 leader_election.cc:302] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c28b93149fc84515a5645a9697ef73bb, f96b510a824d44899a323c0956d721cd; no voters:
I20260927 07:07:07.177618 5826 raft_consensus.cc:2781] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:07.178073 5386 raft_consensus.cc:2454] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c28b93149fc84515a5645a9697ef73bb in term 1.
I20260927 07:07:07.178200 5826 raft_consensus.cc:686] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 1 LEADER]: Becoming Leader. State: Replica: c28b93149fc84515a5645a9697ef73bb, State: Running, Role: LEADER
I20260927 07:07:07.178790 5826 consensus_queue.cc:237] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } }
I20260927 07:07:07.179448 5582 heartbeater.cc:503] Master 127.30.16.125:39935 was elected leader, sending a full tablet report...
I20260927 07:07:07.179724 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.179169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:51183)
0927 07:07:07.179399 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.179521 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 07:07:07.183100 5065 catalog_manager.cc:5495] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb reported cstate change: term changed from 0 to 1, leader changed from <none> to c28b93149fc84515a5645a9697ef73bb (127.30.16.66). New cstate: current_term: 1 leader_uuid: "c28b93149fc84515a5645a9697ef73bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c28b93149fc84515a5645a9697ef73bb" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:07.207676 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.200749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.30.16.66:43105)
0927 07:07:07.200962 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.200967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.201188 (+ 221us) server_negotiation.cc:407] Connection header received
0927 07:07:07.201423 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.201428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.201524 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.201656 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:07.202099 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.202967 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.205231 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.205699 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.205800 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.206340 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:07.207037 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:07.207152 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.207296 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.207368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:07:07.212401 5677 raft_consensus.cc:1261] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Refusing update from remote peer c28b93149fc84515a5645a9697ef73bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:07.213040 5386 raft_consensus.cc:1261] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Refusing update from remote peer c28b93149fc84515a5645a9697ef73bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:07.213280 5835 consensus_queue.cc:1035] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f96b510a824d44899a323c0956d721cd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 35957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:07.213836 5826 consensus_queue.cc:1035] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f3a6d8b860944a79548ab3b133a9c75" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:07.217132 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.216471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:60825)
0927 07:07:07.216767 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.216878 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 07:07:07.217819 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.216515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:34291)
0927 07:07:07.217448 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.217582 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":780}
I20260927 07:07:07.233004 5839 mvcc.cc:205] Tried to move back new op lower bound from 7333858620254597120 to 7333858620130082816. Current Snapshot: MvccSnapshot[applied={T|T < 7333858620254597120}]
I20260927 07:07:07.233527 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.232594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.67:35299)
0927 07:07:07.233177 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.233300 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
I20260927 07:07:07.236500 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.235830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.65:58367)
0927 07:07:07.236136 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.236255 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260927 07:07:07.243151 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.240235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.30.16.66:48773)
0927 07:07:07.242783 (+ 2548us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.242917 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2383}
I20260927 07:07:07.253540 5065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:07.253851 5065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:07.260377 5065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e7413d90bde44999e00ce9fa6053b41: Sending DeleteTablet for 3 replicas of tablet 4d6f4f7a1e0a410798d932a439ddb887
I20260927 07:07:07.261252 5515 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:07 UTC) from {username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:55656
I20260927 07:07:07.261786 5657 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:07 UTC) from {username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:50936
I20260927 07:07:07.262648 5366 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d6f4f7a1e0a410798d932a439ddb887 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:07 UTC) from {username='oryx', principal='oryx/127.30.16.125@KRBTEST.COM'} at 127.0.0.1:49840
I20260927 07:07:07.263984 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5236
I20260927 07:07:07.270799 5849 tablet_replica.cc:330] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: stopping tablet replica
I20260927 07:07:07.271001 5848 tablet_replica.cc:330] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: stopping tablet replica
I20260927 07:07:07.271034 5849 raft_consensus.cc:2229] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:07.271257 5848 raft_consensus.cc:2229] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:07.271490 5849 raft_consensus.cc:2258] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:07.271947 5848 raft_consensus.cc:2258] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:07.272940 5849 ts_tablet_manager.cc:1905] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:07.272946 5848 ts_tablet_manager.cc:1905] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:07.279793 5849 ts_tablet_manager.cc:1918] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:07.279865 5848 ts_tablet_manager.cc:1918] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:07.280009 5848 log.cc:1196] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/4d6f4f7a1e0a410798d932a439ddb887
I20260927 07:07:07.280136 5849 log.cc:1196] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/4d6f4f7a1e0a410798d932a439ddb887
I20260927 07:07:07.280584 5848 ts_tablet_manager.cc:1939] T 4d6f4f7a1e0a410798d932a439ddb887 P c28b93149fc84515a5645a9697ef73bb: Deleting consensus metadata
I20260927 07:07:07.280706 5849 ts_tablet_manager.cc:1939] T 4d6f4f7a1e0a410798d932a439ddb887 P f96b510a824d44899a323c0956d721cd: Deleting consensus metadata
I20260927 07:07:07.281903 5052 catalog_manager.cc:4841] TS c28b93149fc84515a5645a9697ef73bb (127.30.16.66:43105): tablet 4d6f4f7a1e0a410798d932a439ddb887 (table test-table [id=1b1bddc8574c4889855d4e56f93376cd]) successfully deleted
I20260927 07:07:07.281935 5053 catalog_manager.cc:4841] TS f96b510a824d44899a323c0956d721cd (127.30.16.67:35957): tablet 4d6f4f7a1e0a410798d932a439ddb887 (table test-table [id=1b1bddc8574c4889855d4e56f93376cd]) successfully deleted
I20260927 07:07:07.283712 5850 tablet_replica.cc:330] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: stopping tablet replica
I20260927 07:07:07.283937 5850 raft_consensus.cc:2229] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:07.284282 5850 raft_consensus.cc:2258] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:07.285296 5850 ts_tablet_manager.cc:1905] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:07.291414 5850 ts_tablet_manager.cc:1918] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:07.291566 5850 log.cc:1196] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/4d6f4f7a1e0a410798d932a439ddb887
I20260927 07:07:07.292119 5850 ts_tablet_manager.cc:1939] T 4d6f4f7a1e0a410798d932a439ddb887 P 3f3a6d8b860944a79548ab3b133a9c75: Deleting consensus metadata
I20260927 07:07:07.293438 5053 catalog_manager.cc:4841] TS 3f3a6d8b860944a79548ab3b133a9c75 (127.30.16.65:41273): tablet 4d6f4f7a1e0a410798d932a439ddb887 (table test-table [id=1b1bddc8574c4889855d4e56f93376cd]) successfully deleted
I20260927 07:07:07.301268 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.265487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.30.16.65:41273)
0927 07:07:07.265677 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.265683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.283082 (+ 17399us) server_negotiation.cc:407] Connection header received
0927 07:07:07.283292 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.283300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.283417 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.283558 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:07.285872 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.287248 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.289161 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.289927 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.290081 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.293747 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:07.293771 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:07.293795 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:07.293848 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:07.298126 (+ 4278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.298977 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.298988 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.298992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.299156 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.299522 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.299528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.299532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.300068 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:07.300407 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.300582 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.300821 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 07:07:07.302040 5426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:07.424186 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5236
I20260927 07:07:07.442950 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5443
I20260927 07:07:07.457316 5817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.443793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55682 (local address 127.30.16.66:43105)
0927 07:07:07.443963 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.443969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.444058 (+ 89us) server_negotiation.cc:407] Connection header received
0927 07:07:07.444282 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.444286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.444361 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.444484 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:07.445714 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.446533 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.447873 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.448317 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.448454 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.451234 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:07.451249 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:07.451264 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:07.451298 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:07.454366 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.455227 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.455237 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.455241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.455413 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.455787 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.455793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.455797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.456157 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:07.456500 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.456760 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.456980 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:07:07.458127 5575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:07.579106 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5443
I20260927 07:07:07.598273 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5593
I20260927 07:07:07.611594 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.599117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50944 (local address 127.30.16.67:35957)
0927 07:07:07.599284 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:07.599289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:07.599307 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:07:07.599436 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:07.599440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:07.599502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:07.599594 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:07.600872 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.601630 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.602936 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:07.603438 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:07.603573 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:07.605977 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:07.605991 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:07.606033 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:07.606068 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:07.608762 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.609448 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.609456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.609458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.609582 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:07.609919 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:07.609922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:07.609924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:07.610296 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:07.610504 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:07.611169 (+ 665us) server_negotiation.cc:299] Negotiation successful
0927 07:07:07.611328 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:07:07.612260 5717 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:07.637665 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.636822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46250)
0927 07:07:07.637374 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.637494 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":338,"threads_started":1}
I20260927 07:07:07.639065 5855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:07.638715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39821 (local address 127.0.0.1:46256)
0927 07:07:07.638872 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:07.638958 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:39821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 07:07:07.707137 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5593
I20260927 07:07:07.721364 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 4929
I20260927 07:07:07.722781 5096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:07.847160 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 4929
I20260927 07:07:07.868777 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5029
I20260927 07:07:07.869837 5283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:07.985090 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5029
2026-09-27T07:07:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10143 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:08 dist-test-slave-8vqf krb5kdc[5861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:08 dist-test-slave-8vqf krb5kdc[5861](info): set up 2 sockets
Sep 27 07:07:08 dist-test-slave-8vqf krb5kdc[5861](info): commencing operation
krb5kdc: starting...
W20260927 07:07:09.131731 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:07:09 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab.
Entry for principal oryx/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab.
I20260927 07:07:09.362749 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36171
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab
--principal=oryx/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:36171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:09.695370 5874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:09.695683 5874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:09.695844 5874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:09.703210 5874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:09.703323 5874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 07:07:09.703420 5874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:09.703531 5874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:09.703601 5874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:09.703679 5874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:09.714002 5874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:36171
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.126.keytab
--principal=oryx/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:09.715546 5874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:09.716941 5874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:09.729341 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:09.729303 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:09.729267 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:09.730602 5874 server_base.cc:1034] running on GCE node
I20260927 07:07:09.731619 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:09.733271 5874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:09.734679 5874 hybrid_clock.cc:648] HybridClock initialized: now 1790492829734611 us; error 62 us; skew 500 ppm
I20260927 07:07:09.735113 5874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:09 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492829, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:09.738308 5874 init.cc:377] Logged in from keytab as oryx/127.30.16.126@KRBTEST.COM (short username oryx)
I20260927 07:07:09.739964 5874 webserver.cc:460] Webserver started at http://127.30.16.126:34379/ using document root <none> and password file <none>
I20260927 07:07:09.740671 5874 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:09.740760 5874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:09.741055 5874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:09.744302 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "16a01a472f754ee392436f1c902cdb24"
format_stamp: "Formatted at 2026-09-27 07:07:09 on dist-test-slave-8vqf"
server_key: "0c2aab59a1485e079e30df0bb12d1914"
server_key_iv: "2b50c33274e8559bc90a54a8b5a90180"
server_key_version: "encryptionkey@0"
I20260927 07:07:09.745117 5874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "16a01a472f754ee392436f1c902cdb24"
format_stamp: "Formatted at 2026-09-27 07:07:09 on dist-test-slave-8vqf"
server_key: "0c2aab59a1485e079e30df0bb12d1914"
server_key_iv: "2b50c33274e8559bc90a54a8b5a90180"
server_key_version: "encryptionkey@0"
I20260927 07:07:09.750927 5874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 07:07:09.754808 5889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:09.756439 5874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:07:09.756636 5874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "16a01a472f754ee392436f1c902cdb24"
format_stamp: "Formatted at 2026-09-27 07:07:09 on dist-test-slave-8vqf"
server_key: "0c2aab59a1485e079e30df0bb12d1914"
server_key_iv: "2b50c33274e8559bc90a54a8b5a90180"
server_key_version: "encryptionkey@0"
I20260927 07:07:09.756850 5874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:09.795673 5874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:09.800794 5874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:09.801134 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:09.824141 5874 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:36171
I20260927 07:07:09.824160 5950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:36171 every 8 connection(s)
I20260927 07:07:09.825682 5874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:09.830052 5951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:09.835558 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5874
I20260927 07:07:09.835773 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:07:09.836454 30785 external_mini_cluster.cc:1428] Setting key 260081738b62742db41af5219b07333e
I20260927 07:07:09.841625 5951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Bootstrap starting.
I20260927 07:07:09.845513 5951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:09.846841 5951 log.cc:824] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:09.850153 5951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: No bootstrap required, opened a new log
Sep 27 07:07:09 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:09.861037 5951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } }
I20260927 07:07:09.861532 5951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:09.861644 5951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a01a472f754ee392436f1c902cdb24, State: Initialized, Role: FOLLOWER
I20260927 07:07:09.862322 5951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } }
I20260927 07:07:09.862694 5951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:09.862890 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:09.838597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51972 (local address 127.30.16.126:36171)
0927 07:07:09.839480 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:09.839496 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:09.839721 (+ 225us) server_negotiation.cc:407] Connection header received
0927 07:07:09.841637 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:09.841661 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:09.842153 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:09.842504 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:09.844455 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:09.846070 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:09.847964 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:09.848702 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:09.848849 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:09.853612 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:09.853624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:09.853648 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:09.853700 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:09.858012 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:09.858786 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:09.858796 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:09.858806 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:09.859032 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:09.859389 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:09.859398 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:09.859402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:09.860189 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:09.860552 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:09.860885 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 07:07:09.861344 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":414,"threads_started":1}
I20260927 07:07:09.862852 5951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:09.863083 5951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:09.865500 5951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } }
I20260927 07:07:09.866137 5951 leader_election.cc:302] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16a01a472f754ee392436f1c902cdb24; no voters:
I20260927 07:07:09.866802 5951 leader_election.cc:288] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:09.866953 5956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:09.867718 5956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [term 1 LEADER]: Becoming Leader. State: Replica: 16a01a472f754ee392436f1c902cdb24, State: Running, Role: LEADER
I20260927 07:07:09.868366 5956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } }
I20260927 07:07:09.869124 5951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:09.871557 5957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16a01a472f754ee392436f1c902cdb24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } } }
I20260927 07:07:09.871655 5958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16a01a472f754ee392436f1c902cdb24. Latest consensus state: current_term: 1 leader_uuid: "16a01a472f754ee392436f1c902cdb24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16a01a472f754ee392436f1c902cdb24" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36171 } } }
I20260927 07:07:09.871927 5957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:09.872010 5958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:09.873557 5962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:09.882285 5962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:09.892561 5962 catalog_manager.cc:1347] Generated new cluster ID: e1a2d6fdb3384788a0c73f7bdd55cc4d
I20260927 07:07:09.892724 5962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:09.905371 5962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:09.907068 5962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:09.913625 5962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Generated new TSK 0
I20260927 07:07:09.914481 5962 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
I20260927 07:07:09.975790 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36171
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:10.347473 5977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:10.347775 5977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:10.347872 5977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:10.355367 5977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:10.355510 5977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:10.355597 5977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:07:10.355679 5977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:10.366034 5977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:10.367461 5977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:10.368847 5977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:10.383625 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:10.383608 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:10.384171 5985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:10.384788 5977 server_base.cc:1034] running on GCE node
I20260927 07:07:10.385484 5977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:10.386523 5977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:10.387822 5977 hybrid_clock.cc:648] HybridClock initialized: now 1790492830387754 us; error 57 us; skew 500 ppm
I20260927 07:07:10.388170 5977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:10 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492830, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:10.391592 5977 init.cc:377] Logged in from keytab as oryx/127.30.16.65@KRBTEST.COM (short username oryx)
I20260927 07:07:10.393102 5977 webserver.cc:460] Webserver started at http://127.30.16.65:37317/ using document root <none> and password file <none>
I20260927 07:07:10.393793 5977 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:10.393882 5977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:10.394151 5977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:10.397166 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "1ddf79befb6ee4d1dfa0c937bbff1d23"
server_key_iv: "aad5d430cd2b0694b17207d964a8f167"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.397933 5977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "1ddf79befb6ee4d1dfa0c937bbff1d23"
server_key_iv: "aad5d430cd2b0694b17207d964a8f167"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.403338 5977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:07:10.407054 5992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:10.408468 5977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:07:10.408634 5977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "1ddf79befb6ee4d1dfa0c937bbff1d23"
server_key_iv: "aad5d430cd2b0694b17207d964a8f167"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.408876 5977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:10.423516 5977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:10.428176 5977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:10.428478 5977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:10.431432 5977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:10.431527 5977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:10.431710 5977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:10.431795 5977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:07:10 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492830, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:10.455022 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.431963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:45903 (local address 127.30.16.126:36171)
0927 07:07:10.432760 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:10.432767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:10.433948 (+ 1181us) server_negotiation.cc:407] Connection header received
0927 07:07:10.436129 (+ 2181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:10.436135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:10.436232 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:10.436349 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:10.438772 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.439863 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.442680 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.443285 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.443429 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:10.448533 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:10.448537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:10.448541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:10.448575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:10.451562 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:10.452591 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:10.452597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:10.452600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:10.452725 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:10.453176 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:10.453181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:10.453184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:10.453467 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:10.453630 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:10.454399 (+ 769us) server_negotiation.cc:299] Negotiation successful
0927 07:07:10.454661 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":540,"threads_started":1}
I20260927 07:07:10.456655 6002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.432789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.65:45903)
0927 07:07:10.433728 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.433782 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:10.435814 (+ 2032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:10.436660 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:10.436678 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:10.437549 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:10.438532 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:10.438590 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.440032 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.440038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:10.442456 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:10.442468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.443558 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.443565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:10.443814 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:10.444700 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:10.444722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:10.448271 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:10.451700 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:10.451714 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:10.451732 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:10.452454 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:10.452838 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:10.452843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:10.452846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:10.453057 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:10.453611 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:10.453619 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:10.454013 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 07:07:10.454287 (+ 274us) client_negotiation.cc:240] Negotiation successful
0927 07:07:10.454898 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":405,"threads_started":1}
I20260927 07:07:10.480929 6001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.465617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:49839 (local address 127.30.16.126:36171)
0927 07:07:10.465861 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:10.465871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:10.466519 (+ 648us) server_negotiation.cc:407] Connection header received
0927 07:07:10.466811 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:10.466819 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:10.466912 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:10.467031 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:10.468592 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.469389 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.471012 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.471447 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.471596 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:10.474532 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:10.474540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:10.474545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:10.474628 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:10.477383 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:10.478426 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:10.478430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:10.478432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:10.478609 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:10.479173 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:10.479181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:10.479184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:10.479520 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:10.479707 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:10.480333 (+ 626us) server_negotiation.cc:299] Negotiation successful
0927 07:07:10.480563 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:07:10.481066 6070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.465581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.65:49839)
0927 07:07:10.466371 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.466396 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:10.466611 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:10.467068 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:10.467073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:10.467893 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:10.468423 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:10.468436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.469550 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.469555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:10.470851 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:10.470867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:10.471643 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:10.471649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:10.471843 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:10.472695 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:10.472729 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:10.474275 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:10.477534 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:10.477543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:10.477550 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:10.478286 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:10.478740 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:10.478746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:10.478750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:10.478983 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:10.479780 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:10.479787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:10.480025 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 07:07:10.480226 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 07:07:10.480590 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":388,"threads_started":1}
I20260927 07:07:10.492017 5977 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36759
I20260927 07:07:10.492148 6112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36759 every 8 connection(s)
I20260927 07:07:10.493337 5977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:10.495428 6113 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36171
I20260927 07:07:10.495668 6113 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:10.496346 6113 heartbeater.cc:511] Master 127.30.16.126:36171 requested a full tablet report, sending...
I20260927 07:07:10.498347 5906 ts_manager.cc:194] Registered new tserver with Master: 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759)
I20260927 07:07:10.499882 5906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:45903
I20260927 07:07:10.503113 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 5977
I20260927 07:07:10.503347 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:10.504010 30785 external_mini_cluster.cc:1428] Setting key 37f55394d144cefbf58ae31d91d53709
WARNING: no policy specified for oryx/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
I20260927 07:07:10.559279 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36171
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:10.902581 6117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:10.902941 6117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:10.903133 6117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:10.910790 6117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:10.910967 6117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:10.911135 6117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:07:10.911278 6117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:10.921757 6117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:10.923301 6117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:10.924746 6117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:10.939661 6125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:10.939803 6122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:10.939761 6123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:10.940573 6117 server_base.cc:1034] running on GCE node
I20260927 07:07:10.941265 6117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:10.942314 6117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:10.943598 6117 hybrid_clock.cc:648] HybridClock initialized: now 1790492830943540 us; error 32 us; skew 500 ppm
I20260927 07:07:10.943981 6117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:10 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492830, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:10.947621 6117 init.cc:377] Logged in from keytab as oryx/127.30.16.66@KRBTEST.COM (short username oryx)
I20260927 07:07:10.949487 6117 webserver.cc:460] Webserver started at http://127.30.16.66:36425/ using document root <none> and password file <none>
I20260927 07:07:10.950333 6117 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:10.950716 6117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:10.951164 6117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:10.954643 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "352598c644d2454d8d9b2bcd7b30f821"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "a6c1345513d6440c222edd261d45e9b4"
server_key_iv: "e64b195d8e590a1e3e372e3fd21e508d"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.955572 6117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "352598c644d2454d8d9b2bcd7b30f821"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "a6c1345513d6440c222edd261d45e9b4"
server_key_iv: "e64b195d8e590a1e3e372e3fd21e508d"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.962046 6117 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260927 07:07:10.966074 6132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:10.967640 6117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:10.967842 6117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "352598c644d2454d8d9b2bcd7b30f821"
format_stamp: "Formatted at 2026-09-27 07:07:10 on dist-test-slave-8vqf"
server_key: "a6c1345513d6440c222edd261d45e9b4"
server_key_iv: "e64b195d8e590a1e3e372e3fd21e508d"
server_key_version: "encryptionkey@0"
I20260927 07:07:10.968091 6117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:10.982759 6117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:10.987700 6117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:10.988029 6117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:10.991093 6117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:10.991185 6117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:10.991379 6117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:10.991441 6117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492830, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:11.017177 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.991911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:48325 (local address 127.30.16.126:36171)
0927 07:07:10.992398 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:10.992404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:10.993634 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 07:07:10.995976 (+ 2342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:10.995982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:10.996099 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:10.996249 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:10.998975 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.000035 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.001752 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.002353 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.002505 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.008858 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:11.008867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:11.008872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:11.008926 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:11.012350 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.013803 (+ 1453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.013811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.013815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.013981 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.014745 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.014754 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.014758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.015180 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:11.015362 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:11.016524 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0927 07:07:11.016769 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260927 07:07:11.019039 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:10.992540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.66:48325)
0927 07:07:10.993434 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:10.993481 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:10.995577 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:10.996611 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:10.996629 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:10.997579 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:10.998762 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:10.998795 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.000201 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.000206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.001596 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.001609 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.002695 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.002706 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.003049 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.004061 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:11.004109 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:11.008479 (+ 4370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:11.012559 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.012575 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.012610 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.013601 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.014202 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.014206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.014209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.014535 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.015477 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:11.015487 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:11.016074 (+ 587us) client_negotiation.cc:769] Sending connection context
0927 07:07:11.016411 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 07:07:11.017069 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":372,"threads_started":1}
I20260927 07:07:11.039755 6117 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:42555
I20260927 07:07:11.039794 6252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:42555 every 8 connection(s)
I20260927 07:07:11.041110 6117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:11.042451 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6117
I20260927 07:07:11.042670 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:11.043239 30785 external_mini_cluster.cc:1428] Setting key 8ceb1e7f39fc6e260804f70c376fc39e
I20260927 07:07:11.043920 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.026956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:34283 (local address 127.30.16.126:36171)
0927 07:07:11.027401 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:11.027407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:11.028225 (+ 818us) server_negotiation.cc:407] Connection header received
0927 07:07:11.028495 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:11.028500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:11.028580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:11.028676 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:11.030675 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.031750 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.034435 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.034942 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.035124 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.037900 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:11.037913 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:11.037917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:11.037956 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:11.040805 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.041580 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.041587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.041591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.041719 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.042198 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.042203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.042208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.042722 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:11.042878 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:11.043332 (+ 454us) server_negotiation.cc:299] Negotiation successful
0927 07:07:11.043562 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260927 07:07:11.043509 6253 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36171
I20260927 07:07:11.044019 6228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.026996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.66:34283)
0927 07:07:11.028059 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.028083 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:11.028282 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:11.028747 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:11.028754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:11.029779 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:11.030404 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.030427 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.031917 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.031926 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.034246 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.034268 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.035194 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.035203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.035433 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.036300 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:11.036331 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:11.037671 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:11.040942 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.040948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.040954 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.041458 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.041839 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.041844 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.041847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.042040 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.042842 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:11.042847 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:11.043050 (+ 203us) client_negotiation.cc:769] Sending connection context
0927 07:07:11.043225 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 07:07:11.043562 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":725,"threads_started":1}
I20260927 07:07:11.044866 6253 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:11.045790 6253 heartbeater.cc:511] Master 127.30.16.126:36171 requested a full tablet report, sending...
I20260927 07:07:11.047323 5906 ts_manager.cc:194] Registered new tserver with Master: 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555)
I20260927 07:07:11.048120 5906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:48325
WARNING: no policy specified for oryx/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
I20260927 07:07:11.094930 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36171
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:11.432964 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:11.433279 6257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:11.433442 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:11.440920 6257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:11.441051 6257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:11.441193 6257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 07:07:11.441316 6257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:11.451646 6257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.30.16.126:36171
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:11.452999 6257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:11.454401 6257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:11.468304 6262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:11.468281 6263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:11.468532 6265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:11.469087 6257 server_base.cc:1034] running on GCE node
I20260927 07:07:11.469842 6257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:11.471068 6257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:11.472309 6257 hybrid_clock.cc:648] HybridClock initialized: now 1790492831472252 us; error 42 us; skew 500 ppm
I20260927 07:07:11.472666 6257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492831, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:11.476049 6257 init.cc:377] Logged in from keytab as oryx/127.30.16.67@KRBTEST.COM (short username oryx)
I20260927 07:07:11.477524 6257 webserver.cc:460] Webserver started at http://127.30.16.67:44187/ using document root <none> and password file <none>
I20260927 07:07:11.478246 6257 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:11.478379 6257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:11.478773 6257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:11.481805 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "e6ef9684930f4142af5e78fa2626635b"
format_stamp: "Formatted at 2026-09-27 07:07:11 on dist-test-slave-8vqf"
server_key: "58f2adb5355d718ee9db3cfca0a75cf1"
server_key_iv: "1531b5161b7ed3306cc135433de28d4c"
server_key_version: "encryptionkey@0"
I20260927 07:07:11.482702 6257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "e6ef9684930f4142af5e78fa2626635b"
format_stamp: "Formatted at 2026-09-27 07:07:11 on dist-test-slave-8vqf"
server_key: "58f2adb5355d718ee9db3cfca0a75cf1"
server_key_iv: "1531b5161b7ed3306cc135433de28d4c"
server_key_version: "encryptionkey@0"
I20260927 07:07:11.488278 6257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:07:11.492487 6272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:11.494179 6257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 07:07:11.494407 6257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "e6ef9684930f4142af5e78fa2626635b"
format_stamp: "Formatted at 2026-09-27 07:07:11 on dist-test-slave-8vqf"
server_key: "58f2adb5355d718ee9db3cfca0a75cf1"
server_key_iv: "1531b5161b7ed3306cc135433de28d4c"
server_key_version: "encryptionkey@0"
I20260927 07:07:11.494726 6257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:11.502866 6113 heartbeater.cc:503] Master 127.30.16.126:36171 was elected leader, sending a full tablet report...
I20260927 07:07:11.516897 6257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:11.521842 6257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:11.522147 6257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:11.525115 6257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:11.525231 6257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:11.525421 6257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:11.525502 6257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492831, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:11.553567 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.526138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:36847 (local address 127.30.16.126:36171)
0927 07:07:11.526349 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:11.526355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:11.527990 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 07:07:11.531193 (+ 3203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:11.531202 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:11.531326 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:11.531432 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:11.534902 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.535960 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.539446 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.540042 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.540184 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.545775 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:11.545784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:11.545791 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:11.545847 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:11.549473 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.550386 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.550391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.550393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.550526 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.551188 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.551194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.551198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.551673 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:11.551856 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:11.552686 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 07:07:11.553013 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:07:11.554194 6281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.526832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.67:36847)
0927 07:07:11.527780 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.527835 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:11.530765 (+ 2930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:11.532122 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:11.532135 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:11.533328 (+ 1193us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:11.534611 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.534646 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.536193 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.536201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.539254 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.539271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.540545 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.540554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.540888 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.541777 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:11.541809 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:11.545463 (+ 3654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:11.549659 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.549671 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.549691 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.550220 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.550702 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.550708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.550712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.550981 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.551839 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:11.551849 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:11.552256 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 07:07:11.552505 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 07:07:11.552929 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":369,"threads_started":1}
I20260927 07:07:11.576732 6344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.560619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.30.16.67:55715)
0927 07:07:11.561289 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:11.561313 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:11.561450 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:11.561883 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:11.561886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:11.562367 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:11.562774 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.562784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.564131 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.564138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.565924 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:11.565935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.567081 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.567091 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:11.567351 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.568175 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:11.568207 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:11.569649 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:11.573139 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.573147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.573154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.573753 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.574244 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:11.574248 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:11.574252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:11.574449 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:11.575245 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:11.575249 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:11.575487 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 07:07:11.575696 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 07:07:11.576087 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":384,"threads_started":1}
I20260927 07:07:11.577519 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:11.560587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:55715 (local address 127.30.16.126:36171)
0927 07:07:11.560812 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:11.560821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:11.561407 (+ 586us) server_negotiation.cc:407] Connection header received
0927 07:07:11.561569 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:11.561577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:11.561676 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:11.561840 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:11.562930 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.563994 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.566070 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.566771 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.566928 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:11.569839 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:11.569844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:11.569848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:11.569884 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:11.572976 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.573938 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.573946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.573949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.574113 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:11.574634 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:11.574643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:11.574649 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:11.575119 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:11.575276 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:11.577087 (+ 1811us) server_negotiation.cc:299] Negotiation successful
0927 07:07:11.577288 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 07:07:11.589519 6257 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:41283
I20260927 07:07:11.589555 6391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:41283 every 8 connection(s)
I20260927 07:07:11.590952 6257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:11.593199 6392 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36171
I20260927 07:07:11.593442 6392 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:11.594111 6392 heartbeater.cc:511] Master 127.30.16.126:36171 requested a full tablet report, sending...
I20260927 07:07:11.595504 5906 ts_manager.cc:194] Registered new tserver with Master: e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:11.596330 5906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:36847
I20260927 07:07:11.597802 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6257
I20260927 07:07:11.598018 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:11.598491 30785 external_mini_cluster.cc:1428] Setting key 72d8879f1f775ba4c3f116d68a8d76db
I20260927 07:07:11.601692 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:11 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:12.050529 6253 heartbeater.cc:503] Master 127.30.16.126:36171 was elected leader, sending a full tablet report...
I20260927 07:07:12.599116 6392 heartbeater.cc:503] Master 127.30.16.126:36171 was elected leader, sending a full tablet report...
W20260927 07:07:12.720782 6400 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.073s user 0.000s sys 0.015s
W20260927 07:07:12.722240 6141 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:11.641723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.30.16.126:36171)
0927 07:07:11.642039 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:11.642046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:11.642182 (+ 136us) server_negotiation.cc:407] Connection header received
0927 07:07:11.642419 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:11.642423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:11.642487 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:11.642641 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:11.644024 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.644758 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.646052 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:11.646488 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:11.646648 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.721905 (+1075257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51976: BlockingRecv error: recv got EOF from 127.0.0.1:51976 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 27 07:07:12 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492831, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:12.742460 6141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.728400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51992 (local address 127.30.16.126:36171)
0927 07:07:12.728628 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.728632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.728963 (+ 331us) server_negotiation.cc:407] Connection header received
0927 07:07:12.729113 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.729117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.729186 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.729309 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:12.730785 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.731589 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.732797 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.733206 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.733308 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.737527 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:12.737532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:12.737535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:12.737569 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:12.740190 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:12.740821 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:12.740824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:12.740827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:12.740994 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:12.741294 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:12.741297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:12.741299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:12.741563 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:12.741747 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.742062 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.742228 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 07:07:12.746942 5906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:12.753969 5906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:12.791705 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.768088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36304 (local address 127.30.16.66:42555)
0927 07:07:12.781208 (+ 13120us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.781223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.782353 (+ 1130us) server_negotiation.cc:407] Connection header received
0927 07:07:12.782461 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.782469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.782779 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.782987 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:12.783998 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.785463 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.788463 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.790006 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.790117 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.790379 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.791106 (+ 727us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.791355 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12869,"thread_start_us":13620,"threads_started":1}
I20260927 07:07:12.792698 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.767908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42555 (local address 127.0.0.1:36304)
0927 07:07:12.781722 (+ 13814us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.781803 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:12.782045 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:12.783215 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:12.783220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:12.783248 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:12.783812 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.783826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.785640 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.785644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.788253 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.788269 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.790589 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.790596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.790822 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.790876 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 07:07:12.792022 (+ 1146us) client_negotiation.cc:240] Negotiation successful
0927 07:07:12.792234 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13405,"thread_start_us":13902,"threads_started":1}
I20260927 07:07:12.793388 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.768111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36759 (local address 127.0.0.1:56112)
0927 07:07:12.782087 (+ 13976us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.782106 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:12.782226 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:12.783024 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:12.783032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:12.783067 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:12.783693 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.783712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.786831 (+ 3119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.786836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.789022 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.789037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.792454 (+ 3417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.792460 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.792653 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.792702 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 07:07:12.792772 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 07:07:12.792918 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13844,"mutex_wait_us":1,"thread_start_us":14141,"threads_started":1}
I20260927 07:07:12.794375 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.768456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56112 (local address 127.30.16.65:36759)
0927 07:07:12.781686 (+ 13230us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.781698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.782353 (+ 655us) server_negotiation.cc:407] Connection header received
0927 07:07:12.782461 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.782469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.782780 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.783901 (+ 1121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:12.785165 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.786678 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.789238 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.792163 (+ 2925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.793617 (+ 1454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.793851 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.793938 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.794108 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13087,"thread_start_us":12950,"threads_started":1}
I20260927 07:07:12.798031 6183 tablet_service.cc:1467] Processing CreateTablet for tablet fa1f9678eb48417aa06c28be21697f23 (DEFAULT_TABLE table=test-table [id=5ed932e8b4734354ab67216fffc7312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:12.798350 6043 tablet_service.cc:1467] Processing CreateTablet for tablet fa1f9678eb48417aa06c28be21697f23 (DEFAULT_TABLE table=test-table [id=5ed932e8b4734354ab67216fffc7312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:12.799948 6043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1f9678eb48417aa06c28be21697f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:12.799952 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1f9678eb48417aa06c28be21697f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:12.800552 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.782833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:41283 (local address 127.0.0.1:55944)
0927 07:07:12.792944 (+ 10111us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.792963 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:12.793070 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:12.794593 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:12.794599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:12.794629 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:12.794815 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.794826 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.796525 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.796530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.798096 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.798108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.799828 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.799834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.800004 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.800052 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 07:07:12.800143 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 07:07:12.800297 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9991,"thread_start_us":10511,"threads_started":1}
I20260927 07:07:12.801239 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.783156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55944 (local address 127.30.16.67:41283)
0927 07:07:12.793909 (+ 10753us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.793918 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.793948 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:07:12.794050 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.794059 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.794331 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.795010 (+ 679us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:12.795211 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.796402 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.798269 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.799658 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.799818 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.800031 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.800912 (+ 881us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.801035 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10582,"thread_start_us":10636,"threads_started":1}
I20260927 07:07:12.804392 6322 tablet_service.cc:1467] Processing CreateTablet for tablet fa1f9678eb48417aa06c28be21697f23 (DEFAULT_TABLE table=test-table [id=5ed932e8b4734354ab67216fffc7312c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:12.805644 6322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa1f9678eb48417aa06c28be21697f23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:12.816773 6418 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Bootstrap starting.
I20260927 07:07:12.819607 6419 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Bootstrap starting.
I20260927 07:07:12.819620 6417 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Bootstrap starting.
I20260927 07:07:12.823052 6417 tablet_bootstrap.cc:654] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:12.823624 6418 tablet_bootstrap.cc:654] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:12.823627 6419 tablet_bootstrap.cc:654] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:12.824522 6417 log.cc:824] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:12.825047 6419 log.cc:824] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:12.825047 6418 log.cc:824] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:12.833038 6419 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: No bootstrap required, opened a new log
I20260927 07:07:12.833757 6419 ts_tablet_manager.cc:1397] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260927 07:07:12.846014 6418 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: No bootstrap required, opened a new log
I20260927 07:07:12.846010 6417 tablet_bootstrap.cc:492] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: No bootstrap required, opened a new log
I20260927 07:07:12.846364 6417 ts_tablet_manager.cc:1397] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.012s
I20260927 07:07:12.846447 6418 ts_tablet_manager.cc:1397] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.017s
I20260927 07:07:12.849864 6419 raft_consensus.cc:348] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.850575 6419 raft_consensus.cc:374] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:12.850719 6419 raft_consensus.cc:729] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Initialized, Role: FOLLOWER
I20260927 07:07:12.851837 6419 consensus_queue.cc:260] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.857523 6417 raft_consensus.cc:348] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.857523 6418 raft_consensus.cc:348] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.858019 6417 raft_consensus.cc:374] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:12.858023 6418 raft_consensus.cc:374] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:12.858127 6417 raft_consensus.cc:729] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Initialized, Role: FOLLOWER
I20260927 07:07:12.858127 6418 raft_consensus.cc:729] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Initialized, Role: FOLLOWER
I20260927 07:07:12.858860 6417 consensus_queue.cc:260] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.858858 6418 consensus_queue.cc:260] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.860586 6419 ts_tablet_manager.cc:1428] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Time spent starting tablet: real 0.027s user 0.016s sys 0.006s
I20260927 07:07:12.862697 6417 ts_tablet_manager.cc:1428] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
I20260927 07:07:12.864760 6418 ts_tablet_manager.cc:1428] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20260927 07:07:12.934103 6423 raft_consensus.cc:482] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:12.934489 6423 raft_consensus.cc:504] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.936434 6423 leader_election.cc:288] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555), 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759)
I20260927 07:07:12.943291 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.936972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:43247 (local address 127.30.16.66:42555)
0927 07:07:12.937228 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.937237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.937915 (+ 678us) server_negotiation.cc:407] Connection header received
0927 07:07:12.938165 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.938169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.938229 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.938336 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:12.938931 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.939632 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.941536 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.942327 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.942562 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.942642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.942917 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.943042 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:07:12.943457 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.936811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42555 (local address 127.30.16.67:43247)
0927 07:07:12.937787 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.937804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:12.938012 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:12.938345 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:12.938350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:12.938380 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:12.938782 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.938792 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.939766 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.939772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.941376 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.941388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.942575 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.942582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.942745 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.942770 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:07:12.942882 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 07:07:12.943012 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":44,"thread_start_us":660,"threads_started":1}
I20260927 07:07:12.943948 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.937229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:48003 (local address 127.30.16.65:36759)
0927 07:07:12.937465 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:12.937472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:12.938271 (+ 799us) server_negotiation.cc:407] Connection header received
0927 07:07:12.938519 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:12.938523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:12.938648 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:12.938750 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:12.939376 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.940087 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.941962 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.942916 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.943122 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.943208 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:12.943602 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 07:07:12.943733 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 07:07:12.944736 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:12.936969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36759 (local address 127.30.16.67:48003)
0927 07:07:12.938148 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:12.938164 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:12.938336 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:12.938795 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:12.938802 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:12.938827 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:12.939174 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.939183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.940245 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.940252 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.941774 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:12.941794 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:12.943168 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:12.943175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:12.943347 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:12.943371 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 07:07:12.944170 (+ 799us) client_negotiation.cc:240] Negotiation successful
0927 07:07:12.944313 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":2,"thread_start_us":867,"threads_started":1}
I20260927 07:07:12.944645 6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1f9678eb48417aa06c28be21697f23" candidate_uuid: "e6ef9684930f4142af5e78fa2626635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352598c644d2454d8d9b2bcd7b30f821" is_pre_election: true
I20260927 07:07:12.945274 6203 raft_consensus.cc:2454] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ef9684930f4142af5e78fa2626635b in term 0.
I20260927 07:07:12.945852 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1f9678eb48417aa06c28be21697f23" candidate_uuid: "e6ef9684930f4142af5e78fa2626635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" is_pre_election: true
I20260927 07:07:12.946053 6273 leader_election.cc:302] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, e6ef9684930f4142af5e78fa2626635b; no voters:
I20260927 07:07:12.946388 6067 raft_consensus.cc:2454] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6ef9684930f4142af5e78fa2626635b in term 0.
I20260927 07:07:12.946583 6423 raft_consensus.cc:2781] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:12.946738 6423 raft_consensus.cc:482] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:12.946950 6423 raft_consensus.cc:3037] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:12.949273 6423 raft_consensus.cc:504] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.950018 6423 leader_election.cc:288] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [CANDIDATE]: Term 1 election: Requested vote from peers 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555), 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759)
I20260927 07:07:12.950624 6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1f9678eb48417aa06c28be21697f23" candidate_uuid: "e6ef9684930f4142af5e78fa2626635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352598c644d2454d8d9b2bcd7b30f821"
I20260927 07:07:12.950683 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa1f9678eb48417aa06c28be21697f23" candidate_uuid: "e6ef9684930f4142af5e78fa2626635b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
I20260927 07:07:12.950896 6067 raft_consensus.cc:3037] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:12.950881 6203 raft_consensus.cc:3037] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:12.954187 6067 raft_consensus.cc:2454] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ef9684930f4142af5e78fa2626635b in term 1.
I20260927 07:07:12.954187 6203 raft_consensus.cc:2454] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6ef9684930f4142af5e78fa2626635b in term 1.
I20260927 07:07:12.954699 6273 leader_election.cc:302] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e, e6ef9684930f4142af5e78fa2626635b; no voters:
I20260927 07:07:12.955055 6423 raft_consensus.cc:2781] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:12.955504 6423 raft_consensus.cc:686] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 1 LEADER]: Becoming Leader. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Running, Role: LEADER
I20260927 07:07:12.956195 6423 consensus_queue.cc:237] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:12.961170 5904 catalog_manager.cc:5495] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b reported cstate change: term changed from 0 to 1, leader changed from <none> to e6ef9684930f4142af5e78fa2626635b (127.30.16.67). New cstate: current_term: 1 leader_uuid: "e6ef9684930f4142af5e78fa2626635b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } health_report { overall_health: HEALTHY } } }
W20260927 07:07:12.996865 6114 tablet.cc:2369] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:13.019150 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.014175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.30.16.67:41283)
0927 07:07:13.014371 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.014375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.014392 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:07:13.014485 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.014488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.014589 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.014712 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:13.015033 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.015734 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.017027 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.017437 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.017527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.017918 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.018480 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.018676 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.018819 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.018901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:07:13.027920 6067 raft_consensus.cc:1261] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Refusing update from remote peer e6ef9684930f4142af5e78fa2626635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.028149 6203 raft_consensus.cc:1261] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Refusing update from remote peer e6ef9684930f4142af5e78fa2626635b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.029093 6428 consensus_queue.cc:1035] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.029506 6423 consensus_queue.cc:1035] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [LEADER]: Connected to new peer: Peer: permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.042778 6431 mvcc.cc:205] Tried to move back new op lower bound from 7333858644072140800 to 7333858643793317888. Current Snapshot: MvccSnapshot[applied={T|T < 7333858644072140800}]
W20260927 07:07:13.046245 6254 tablet.cc:2369] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:13.059610 6432 mvcc.cc:205] Tried to move back new op lower bound from 7333858644072140800 to 7333858643793317888. Current Snapshot: MvccSnapshot[applied={T|T < 7333858644072140800}]
I20260927 07:07:13.069370 5906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:13.069952 5906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:13.073797 5906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Sending DeleteTablet for 3 replicas of tablet fa1f9678eb48417aa06c28be21697f23
I20260927 07:07:13.075197 6183 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1f9678eb48417aa06c28be21697f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:36304
I20260927 07:07:13.075299 6322 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1f9678eb48417aa06c28be21697f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55944
I20260927 07:07:13.075580 6043 tablet_service.cc:1514] Processing DeleteTablet for tablet fa1f9678eb48417aa06c28be21697f23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56112
I20260927 07:07:13.079829 5906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:13.081193 5906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:13.086309 6442 tablet_replica.cc:330] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: stopping tablet replica
I20260927 07:07:13.086984 6442 raft_consensus.cc:2229] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:13.087846 6444 tablet_replica.cc:330] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: stopping tablet replica
I20260927 07:07:13.088570 6444 raft_consensus.cc:2229] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:13.089279 6444 raft_consensus.cc:2258] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.090461 6443 tablet_replica.cc:330] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: stopping tablet replica
I20260927 07:07:13.090963 6442 raft_consensus.cc:2258] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.091588 6042 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3e0bb69474c58862135751957c818 (DEFAULT_TABLE table=test-table [id=6098cbc88a8149428e8bfe54502c4e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.092176 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3e0bb69474c58862135751957c818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.092054 6322 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3e0bb69474c58862135751957c818 (DEFAULT_TABLE table=test-table [id=6098cbc88a8149428e8bfe54502c4e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.092653 6322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3e0bb69474c58862135751957c818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.093467 6183 tablet_service.cc:1467] Processing CreateTablet for tablet e9b3e0bb69474c58862135751957c818 (DEFAULT_TABLE table=test-table [id=6098cbc88a8149428e8bfe54502c4e91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.093979 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9b3e0bb69474c58862135751957c818. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.099090 6417 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Bootstrap starting.
I20260927 07:07:13.100765 6418 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Bootstrap starting.
I20260927 07:07:13.100837 6417 tablet_bootstrap.cc:654] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.102831 6418 tablet_bootstrap.cc:654] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.103454 6444 ts_tablet_manager.cc:1905] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.108541 6443 raft_consensus.cc:2229] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:13.109860 6417 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: No bootstrap required, opened a new log
I20260927 07:07:13.110117 6417 ts_tablet_manager.cc:1397] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20260927 07:07:13.110754 6444 ts_tablet_manager.cc:1918] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:13.110965 6444 log.cc:1196] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/fa1f9678eb48417aa06c28be21697f23
I20260927 07:07:13.111209 6417 raft_consensus.cc:348] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.111554 6444 ts_tablet_manager.cc:1939] T fa1f9678eb48417aa06c28be21697f23 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting consensus metadata
I20260927 07:07:13.111658 6417 raft_consensus.cc:374] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.111821 6417 raft_consensus.cc:729] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.112254 6417 consensus_queue.cc:260] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.113557 5893 catalog_manager.cc:4841] TS 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759): tablet fa1f9678eb48417aa06c28be21697f23 (table test-table [id=5ed932e8b4734354ab67216fffc7312c]) successfully deleted
I20260927 07:07:13.113670 6442 ts_tablet_manager.cc:1905] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.114974 6417 ts_tablet_manager.cc:1428] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 07:07:13.115099 6443 raft_consensus.cc:2258] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.118212 6443 ts_tablet_manager.cc:1905] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.122452 6419 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Bootstrap starting.
I20260927 07:07:13.124187 6443 ts_tablet_manager.cc:1918] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:13.124449 6443 log.cc:1196] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/fa1f9678eb48417aa06c28be21697f23
I20260927 07:07:13.124583 6419 tablet_bootstrap.cc:654] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.125171 6443 ts_tablet_manager.cc:1939] T fa1f9678eb48417aa06c28be21697f23 P 352598c644d2454d8d9b2bcd7b30f821: Deleting consensus metadata
I20260927 07:07:13.126143 6418 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: No bootstrap required, opened a new log
I20260927 07:07:13.126300 6418 ts_tablet_manager.cc:1397] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.005s
I20260927 07:07:13.127074 5890 catalog_manager.cc:4841] TS 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555): tablet fa1f9678eb48417aa06c28be21697f23 (table test-table [id=5ed932e8b4734354ab67216fffc7312c]) successfully deleted
I20260927 07:07:13.127171 6418 raft_consensus.cc:348] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.127398 6418 raft_consensus.cc:374] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.127514 6418 raft_consensus.cc:729] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.127797 6418 consensus_queue.cc:260] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.128535 6418 ts_tablet_manager.cc:1428] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 07:07:13.129594 6419 tablet_bootstrap.cc:492] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: No bootstrap required, opened a new log
I20260927 07:07:13.129756 6419 ts_tablet_manager.cc:1397] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 07:07:13.131120 6419 raft_consensus.cc:348] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.131424 6419 raft_consensus.cc:374] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.131565 6419 raft_consensus.cc:729] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.131928 6419 consensus_queue.cc:260] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.132735 6419 ts_tablet_manager.cc:1428] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 07:07:13.136868 6442 ts_tablet_manager.cc:1918] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:13.137046 6442 log.cc:1196] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/fa1f9678eb48417aa06c28be21697f23
I20260927 07:07:13.137576 6442 ts_tablet_manager.cc:1939] T fa1f9678eb48417aa06c28be21697f23 P e6ef9684930f4142af5e78fa2626635b: Deleting consensus metadata
I20260927 07:07:13.139140 5890 catalog_manager.cc:4841] TS e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283): tablet fa1f9678eb48417aa06c28be21697f23 (table test-table [id=5ed932e8b4734354ab67216fffc7312c]) successfully deleted
I20260927 07:07:13.191464 6425 raft_consensus.cc:482] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:13.191696 6425 raft_consensus.cc:504] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.193321 6425 leader_election.cc:288] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:13.201812 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.193934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:40515 (local address 127.30.16.65:36759)
0927 07:07:13.194142 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.194150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.195174 (+ 1024us) server_negotiation.cc:407] Connection header received
0927 07:07:13.195448 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.195455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.195531 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.195597 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:13.196408 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.197329 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.199287 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.200543 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.200709 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.200786 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.201224 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.201531 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:07:13.202013 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.194188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:37855 (local address 127.30.16.67:41283)
0927 07:07:13.194359 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.194365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.194913 (+ 548us) server_negotiation.cc:407] Connection header received
0927 07:07:13.195188 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.195193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.195269 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.195367 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:13.195898 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.196939 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.199622 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.200898 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.201043 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.201111 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.201625 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.201732 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:07:13.202631 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.193712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36759 (local address 127.30.16.66:40515)
0927 07:07:13.195075 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.195093 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.195276 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.195799 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.195803 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.195867 (+ 64us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:13.196236 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.196253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.197459 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.197464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.199145 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.199156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.200824 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.200831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.201023 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.201064 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.202011 (+ 947us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.202159 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"mutex_wait_us":27,"thread_start_us":689,"threads_started":1}
I20260927 07:07:13.202802 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.193616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:41283 (local address 127.30.16.66:37855)
0927 07:07:13.194791 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.194817 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.195030 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.195396 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.195401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.195427 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:13.195733 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.195741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.197071 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.197077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.199464 (+ 2387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.199478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.201260 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.201267 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.201438 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.201467 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.202182 (+ 715us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.202342 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"mutex_wait_us":4,"thread_start_us":1146,"threads_started":1}
I20260927 07:07:13.203704 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3e0bb69474c58862135751957c818" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" is_pre_election: true
I20260927 07:07:13.203706 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3e0bb69474c58862135751957c818" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b" is_pre_election: true
I20260927 07:07:13.203974 6067 raft_consensus.cc:2454] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:13.204157 6347 raft_consensus.cc:2454] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:13.204509 6136 leader_election.cc:302] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e; no voters:
I20260927 07:07:13.204933 6425 raft_consensus.cc:2781] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:13.205075 6425 raft_consensus.cc:482] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:13.205174 6425 raft_consensus.cc:3037] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.207386 6425 raft_consensus.cc:504] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.208065 6425 leader_election.cc:288] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Requested vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:13.208489 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3e0bb69474c58862135751957c818" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
I20260927 07:07:13.208573 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9b3e0bb69474c58862135751957c818" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b"
I20260927 07:07:13.208705 6067 raft_consensus.cc:3037] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.208791 6347 raft_consensus.cc:3037] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.210805 6067 raft_consensus.cc:2454] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:13.211222 6136 leader_election.cc:302] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e; no voters:
I20260927 07:07:13.211516 6425 raft_consensus.cc:2781] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:13.212036 6425 raft_consensus.cc:686] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Becoming Leader. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Running, Role: LEADER
I20260927 07:07:13.212126 6347 raft_consensus.cc:2454] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:13.212770 6425 consensus_queue.cc:237] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.216606 5904 catalog_manager.cc:5495] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "352598c644d2454d8d9b2bcd7b30f821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } health_report { overall_health: UNKNOWN } } }
Sep 27 07:07:13 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492829, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.126@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:13.282675 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.261197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36171 (local address 127.0.0.1:52008)
0927 07:07:13.261916 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.261944 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.262117 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.262586 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.262590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.263328 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:13.263588 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.263597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.264796 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.264802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.265778 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.265789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.266530 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.266535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.266768 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.274300 (+ 7532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:13.274322 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:13.276836 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:13.280063 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:13.280077 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:13.280085 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:13.280518 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:13.280873 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:13.280879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:13.280882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:13.281004 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:13.281523 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:13.281532 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:13.281698 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.281864 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.282162 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":385,"threads_started":1}
I20260927 07:07:13.282709 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.261211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52008 (local address 127.30.16.126:36171)
0927 07:07:13.262300 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.262304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.262320 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:13.262358 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.262361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.262410 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.262538 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:13.263791 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.264647 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.265916 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.266338 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.266476 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.277093 (+ 10617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:13.277099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:13.277102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:13.277132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:13.279887 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:13.280641 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:13.280649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:13.280652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:13.280768 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:13.281103 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:13.281111 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:13.281114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:13.281384 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:13.281541 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.281949 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.282305 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":968,"thread_start_us":766,"threads_started":1}
I20260927 07:07:13.287870 5904 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:52008:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 07:07:13.288908 5904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:13.296123 6183 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb9743db79849fc99356828a86356d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b0cea1396e94754818272540fc832cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:07:13.296151 6042 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb9743db79849fc99356828a86356d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b0cea1396e94754818272540fc832cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:07:13.296679 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb9743db79849fc99356828a86356d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.296689 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb9743db79849fc99356828a86356d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.302587 6418 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: Bootstrap starting.
I20260927 07:07:13.303778 6322 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb9743db79849fc99356828a86356d2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b0cea1396e94754818272540fc832cf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 07:07:13.304327 6322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb9743db79849fc99356828a86356d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.304320 6418 tablet_bootstrap.cc:654] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.305842 6417 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: Bootstrap starting.
I20260927 07:07:13.307907 6417 tablet_bootstrap.cc:654] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.312382 6417 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: No bootstrap required, opened a new log
I20260927 07:07:13.312608 6417 ts_tablet_manager.cc:1397] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 07:07:13.313369 6417 raft_consensus.cc:348] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.313671 6417 raft_consensus.cc:374] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.313762 6417 raft_consensus.cc:729] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.314013 6417 consensus_queue.cc:260] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.314340 6419 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: Bootstrap starting.
I20260927 07:07:13.314455 6424 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.314888 6424 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.316567 6419 tablet_bootstrap.cc:654] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.317353 6418 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: No bootstrap required, opened a new log
I20260927 07:07:13.317556 6418 ts_tablet_manager.cc:1397] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260927 07:07:13.318434 6418 raft_consensus.cc:348] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.318737 6418 raft_consensus.cc:374] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.318825 6418 raft_consensus.cc:729] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.319039 6417 ts_tablet_manager.cc:1428] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260927 07:07:13.319095 6418 consensus_queue.cc:260] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.319742 6418 ts_tablet_manager.cc:1428] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:13.319571 6425 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.319892 6425 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.320017 6419 tablet_bootstrap.cc:492] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: No bootstrap required, opened a new log
I20260927 07:07:13.320204 6419 ts_tablet_manager.cc:1397] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 07:07:13.321049 6419 raft_consensus.cc:348] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.321400 6419 raft_consensus.cc:374] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.321498 6419 raft_consensus.cc:729] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.321774 6419 consensus_queue.cc:260] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.322322 6437 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.323446 6437 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.323836 6419 ts_tablet_manager.cc:1428] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
W20260927 07:07:13.343844 6393 tablet.cc:2369] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:13.444550 6424 raft_consensus.cc:482] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:13.444792 6424 raft_consensus.cc:504] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.446280 6424 leader_election.cc:288] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:13.454731 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.447505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:41283 (local address 127.30.16.65:52461)
0927 07:07:13.448279 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.448305 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.448482 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.448818 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.448822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.448847 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:13.449203 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.449211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.450240 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.450248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.452159 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.452173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.453814 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.453819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.453977 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.454002 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.454137 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.454274 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"spinlock_wait_cycles":2304,"thread_start_us":557,"threads_started":1}
I20260927 07:07:13.455500 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.447867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:52461 (local address 127.30.16.67:41283)
0927 07:07:13.448058 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.448064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.448392 (+ 328us) server_negotiation.cc:407] Connection header received
0927 07:07:13.448628 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.448637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.448701 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.448792 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:13.449356 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.450101 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.452399 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.453601 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.453785 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.453865 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.454163 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.454304 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:07:13.455809 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.446740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42555 (local address 127.30.16.65:39485)
0927 07:07:13.447459 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.447484 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.447660 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.448952 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.448958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.449003 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:13.449479 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.449493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.451118 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.451124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.453136 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.453150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.455027 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.455036 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.455224 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.455259 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.455333 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.455484 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":350,"threads_started":1}
I20260927 07:07:13.456391 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb9743db79849fc99356828a86356d2" candidate_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b" is_pre_election: true
I20260927 07:07:13.456660 6347 raft_consensus.cc:2454] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9971348bc93b4bff88f8b74f83f9ae1e in term 0.
I20260927 07:07:13.456796 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.446812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:39485 (local address 127.30.16.66:42555)
0927 07:07:13.448590 (+ 1778us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.448598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.448619 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:07:13.448697 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.448702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.448780 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.449659 (+ 879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:13.449841 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.450980 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.453304 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.454596 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.456218 (+ 1622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.456289 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.456344 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.456504 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":1534,"threads_started":1}
I20260927 07:07:13.457212 5993 leader_election.cc:302] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9971348bc93b4bff88f8b74f83f9ae1e, e6ef9684930f4142af5e78fa2626635b; no voters:
I20260927 07:07:13.457440 6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb9743db79849fc99356828a86356d2" candidate_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352598c644d2454d8d9b2bcd7b30f821" is_pre_election: true
I20260927 07:07:13.457682 6424 raft_consensus.cc:2781] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:13.457665 6203 raft_consensus.cc:2454] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9971348bc93b4bff88f8b74f83f9ae1e in term 0.
I20260927 07:07:13.457813 6424 raft_consensus.cc:482] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:13.457918 6424 raft_consensus.cc:3037] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.460330 6424 raft_consensus.cc:504] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.461022 6424 leader_election.cc:288] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [CANDIDATE]: Term 1 election: Requested vote from peers 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:13.461531 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb9743db79849fc99356828a86356d2" candidate_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b"
I20260927 07:07:13.461562 6203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb9743db79849fc99356828a86356d2" candidate_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "352598c644d2454d8d9b2bcd7b30f821"
I20260927 07:07:13.461795 6203 raft_consensus.cc:3037] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.461817 6347 raft_consensus.cc:3037] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:13.464130 6347 raft_consensus.cc:2454] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9971348bc93b4bff88f8b74f83f9ae1e in term 1.
I20260927 07:07:13.464529 5993 leader_election.cc:302] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9971348bc93b4bff88f8b74f83f9ae1e, e6ef9684930f4142af5e78fa2626635b; no voters:
I20260927 07:07:13.464811 6424 raft_consensus.cc:2781] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:13.464907 6203 raft_consensus.cc:2454] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9971348bc93b4bff88f8b74f83f9ae1e in term 1.
I20260927 07:07:13.465889 6424 raft_consensus.cc:686] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 LEADER]: Becoming Leader. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Running, Role: LEADER
I20260927 07:07:13.466346 6424 consensus_queue.cc:237] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.469039 6465 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: TxnStatusTablet state changed. Reason: New leader 9971348bc93b4bff88f8b74f83f9ae1e. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.469446 6465 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:07:13.470709 5904 catalog_manager.cc:5495] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65). New cstate: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:13.471064 6467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 07:07:13.471298 6467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 07:07:13.530464 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.523124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.30.16.65:36759)
0927 07:07:13.523309 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.523315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.523637 (+ 322us) server_negotiation.cc:407] Connection header received
0927 07:07:13.523962 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.523968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.524090 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.524269 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:13.524681 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.525493 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.527464 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.528191 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.528327 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.528965 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.529703 (+ 738us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.529865 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.530031 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.530122 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:07:13.530488 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.523124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36759 (local address 127.0.0.1:56120)
0927 07:07:13.523550 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.523572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.523747 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.524296 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.524300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.524313 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:07:13.524541 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.524553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.525634 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.525641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.527277 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.527292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.528388 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.528396 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.528668 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.528758 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.529874 (+ 1116us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.529898 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.529993 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.530157 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260927 07:07:13.535027 6203 raft_consensus.cc:1261] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Refusing update from remote peer 9971348bc93b4bff88f8b74f83f9ae1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.535399 6347 raft_consensus.cc:1261] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Refusing update from remote peer 9971348bc93b4bff88f8b74f83f9ae1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.535933 6465 consensus_queue.cc:1035] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.536664 6424 consensus_queue.cc:1035] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.548822 6437 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: TxnStatusTablet state changed. Reason: New leader 9971348bc93b4bff88f8b74f83f9ae1e. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.549199 6437 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.551729 6424 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.552052 6424 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:07:13.552380 6433 mvcc.cc:205] Tried to move back new op lower bound from 7333858646151499776 to 7333858645883084800. Current Snapshot: MvccSnapshot[applied={T|T < 7333858646151499776}]
I20260927 07:07:13.559589 6425 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: TxnStatusTablet state changed. Reason: New leader 9971348bc93b4bff88f8b74f83f9ae1e. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.559944 6425 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.561949 6465 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.562232 6465 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 9971348bc93b4bff88f8b74f83f9ae1e: This TxnStatusTablet replica's current role is: LEADER
I20260927 07:07:13.562212 6437 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.562469 6437 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P e6ef9684930f4142af5e78fa2626635b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.563593 6425 tablet_replica.cc:439] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } } }
I20260927 07:07:13.563953 6425 tablet_replica.cc:442] T 9fb9743db79849fc99356828a86356d2 P 352598c644d2454d8d9b2bcd7b30f821: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 07:07:13.577178 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.569884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.30.16.66:42555)
0927 07:07:13.570076 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.570085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.570588 (+ 503us) server_negotiation.cc:407] Connection header received
0927 07:07:13.570805 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.570809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.570909 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.571047 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:13.571564 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.572365 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.574502 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.575228 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.575341 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.575925 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.576505 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.576622 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.576755 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.576823 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:07:13.594455 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.588129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:47407 (local address 127.30.16.65:36759)
0927 07:07:13.588294 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.588299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.588804 (+ 505us) server_negotiation.cc:407] Connection header received
0927 07:07:13.589015 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.589021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.589103 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.589229 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:13.589863 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.590646 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.592582 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.593123 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.593273 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.593708 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.593805 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.593908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.594018 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.594098 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:07:13.594514 6477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.587966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36759 (local address 127.30.16.66:47407)
0927 07:07:13.588698 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.588713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.588879 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.589405 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.589410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.589422 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:07:13.589701 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.589709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.590773 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.590782 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.592433 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.592450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.593303 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.593310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.593461 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.593514 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.593908 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.593922 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.594000 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.594143 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":467,"threads_started":1}
I20260927 07:07:13.605140 6067 raft_consensus.cc:1261] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.605490 6347 raft_consensus.cc:1261] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:13.605806 6450 consensus_queue.cc:1035] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.606247 6425 consensus_queue.cc:1035] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:13.650175 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.643719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:57519 (local address 127.30.16.66:42555)
0927 07:07:13.643997 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.644005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.644509 (+ 504us) server_negotiation.cc:407] Connection header received
0927 07:07:13.644716 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.644721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.644805 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.644927 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:13.645528 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.646538 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.648273 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.648707 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.648852 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.649338 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.649506 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.649573 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.649790 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.649869 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:07:13.651021 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.643405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42555 (local address 127.30.16.65:57519)
0927 07:07:13.644409 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:13.644426 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:13.644583 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:13.644943 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:13.644947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:13.644958 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 07:07:13.645339 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.645350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.646719 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.646725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.648127 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:13.648139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.648890 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.648895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:13.649040 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.649125 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 07:07:13.649656 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 07:07:13.649666 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:07:13.650425 (+ 759us) client_negotiation.cc:240] Negotiation successful
0927 07:07:13.650590 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":680,"threads_started":1}
I20260927 07:07:13.693833 5904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:13.694123 5904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:13.697890 5904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Sending DeleteTablet for 3 replicas of tablet e9b3e0bb69474c58862135751957c818
I20260927 07:07:13.699157 6042 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b3e0bb69474c58862135751957c818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56112
I20260927 07:07:13.699965 6183 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b3e0bb69474c58862135751957c818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:36304
I20260927 07:07:13.700249 6322 tablet_service.cc:1514] Processing DeleteTablet for tablet e9b3e0bb69474c58862135751957c818 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:13 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55944
I20260927 07:07:13.700773 6488 tablet_replica.cc:330] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: stopping tablet replica
I20260927 07:07:13.700855 6487 tablet_replica.cc:330] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: stopping tablet replica
I20260927 07:07:13.701043 6487 raft_consensus.cc:2229] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:13.701313 6487 raft_consensus.cc:2258] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.702499 6487 ts_tablet_manager.cc:1905] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.710660 6487 ts_tablet_manager.cc:1918] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:13.710834 6487 log.cc:1196] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/e9b3e0bb69474c58862135751957c818
I20260927 07:07:13.711012 6488 raft_consensus.cc:2229] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:13.711290 6487 ts_tablet_manager.cc:1939] T e9b3e0bb69474c58862135751957c818 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting consensus metadata
I20260927 07:07:13.711799 6488 raft_consensus.cc:2258] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.712473 5893 catalog_manager.cc:4841] TS 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759): tablet e9b3e0bb69474c58862135751957c818 (table test-table [id=6098cbc88a8149428e8bfe54502c4e91]) successfully deleted
I20260927 07:07:13.713150 6488 ts_tablet_manager.cc:1905] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.718319 6489 tablet_replica.cc:330] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: stopping tablet replica
I20260927 07:07:13.718592 6489 raft_consensus.cc:2229] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:13.718842 6489 raft_consensus.cc:2258] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:13.722054 6489 ts_tablet_manager.cc:1905] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:13.722826 6488 ts_tablet_manager.cc:1918] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:13.723026 6488 log.cc:1196] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/e9b3e0bb69474c58862135751957c818
I20260927 07:07:13.723557 6488 ts_tablet_manager.cc:1939] T e9b3e0bb69474c58862135751957c818 P 352598c644d2454d8d9b2bcd7b30f821: Deleting consensus metadata
I20260927 07:07:13.724808 5890 catalog_manager.cc:4841] TS 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555): tablet e9b3e0bb69474c58862135751957c818 (table test-table [id=6098cbc88a8149428e8bfe54502c4e91]) successfully deleted
I20260927 07:07:13.729336 6489 ts_tablet_manager.cc:1918] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:13.729481 6489 log.cc:1196] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/e9b3e0bb69474c58862135751957c818
I20260927 07:07:13.729945 6489 ts_tablet_manager.cc:1939] T e9b3e0bb69474c58862135751957c818 P e6ef9684930f4142af5e78fa2626635b: Deleting consensus metadata
I20260927 07:07:13.731215 5890 catalog_manager.cc:4841] TS e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283): tablet e9b3e0bb69474c58862135751957c818 (table test-table [id=6098cbc88a8149428e8bfe54502c4e91]) successfully deleted
Sep 27 07:07:13 dist-test-slave-8vqf krb5kdc[5861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 07:07:13 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:13 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:07:13.794113 6458 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:13.780526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.30.16.126:36171)
0927 07:07:13.780728 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.780737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.781203 (+ 466us) server_negotiation.cc:407] Connection header received
0927 07:07:13.781404 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.781411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.781498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.781610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:13.783349 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.784512 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.786276 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.787040 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.787172 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.793827 (+ 6655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52022: BlockingRecv error: recv got EOF from 127.0.0.1:52022 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 07:07:13 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM
I20260927 07:07:13.816507 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:13.797964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.30.16.126:36171)
0927 07:07:13.798157 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:13.798163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:13.798182 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:07:13.798245 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:13.798251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:13.798327 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:13.798433 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:13.800102 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.801315 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.803213 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:13.803853 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:13.803995 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:13.809084 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:13.809094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:13.809101 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:13.809155 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:13.812946 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:13.813852 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:13.813858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:13.813862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:13.814043 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:13.814582 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:13.814589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:13.814592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:13.815229 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:13.815506 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:13.815810 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 07:07:13.816073 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 07:07:13.821252 5904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:13.821954 5904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:13.831693 6042 tablet_service.cc:1467] Processing CreateTablet for tablet 93c6837d12c44b4daf550892e00daa1d (DEFAULT_TABLE table=test-table [id=e16681a6b6754222a3e33a5fee6eccb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.832244 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c6837d12c44b4daf550892e00daa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.835013 6183 tablet_service.cc:1467] Processing CreateTablet for tablet 93c6837d12c44b4daf550892e00daa1d (DEFAULT_TABLE table=test-table [id=e16681a6b6754222a3e33a5fee6eccb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.835469 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c6837d12c44b4daf550892e00daa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.838984 6322 tablet_service.cc:1467] Processing CreateTablet for tablet 93c6837d12c44b4daf550892e00daa1d (DEFAULT_TABLE table=test-table [id=e16681a6b6754222a3e33a5fee6eccb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:13.839416 6322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c6837d12c44b4daf550892e00daa1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:13.859200 6502 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Bootstrap starting.
I20260927 07:07:13.861109 6502 tablet_bootstrap.cc:654] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.875289 6502 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: No bootstrap required, opened a new log
I20260927 07:07:13.875487 6502 ts_tablet_manager.cc:1397] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20260927 07:07:13.876334 6505 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Bootstrap starting.
I20260927 07:07:13.876348 6502 raft_consensus.cc:348] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.876644 6502 raft_consensus.cc:374] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.876740 6502 raft_consensus.cc:729] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.877023 6502 consensus_queue.cc:260] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.878579 6502 ts_tablet_manager.cc:1428] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 07:07:13.880882 6505 tablet_bootstrap.cc:654] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.881337 6504 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Bootstrap starting.
I20260927 07:07:13.883100 6504 tablet_bootstrap.cc:654] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:13.886510 6504 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: No bootstrap required, opened a new log
I20260927 07:07:13.886714 6504 ts_tablet_manager.cc:1397] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 07:07:13.887640 6504 raft_consensus.cc:348] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.887905 6504 raft_consensus.cc:374] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.888007 6504 raft_consensus.cc:729] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.888305 6504 consensus_queue.cc:260] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.888972 6504 ts_tablet_manager.cc:1428] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:13.891202 6505 tablet_bootstrap.cc:492] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: No bootstrap required, opened a new log
I20260927 07:07:13.891398 6505 ts_tablet_manager.cc:1397] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260927 07:07:13.892257 6505 raft_consensus.cc:348] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.892540 6505 raft_consensus.cc:374] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:13.892642 6505 raft_consensus.cc:729] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Initialized, Role: FOLLOWER
I20260927 07:07:13.893042 6505 consensus_queue.cc:260] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:13.893651 6505 ts_tablet_manager.cc:1428] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260927 07:07:14.134021 6484 raft_consensus.cc:482] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:14.134298 6484 raft_consensus.cc:504] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.135195 6484 leader_election.cc:288] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:14.135812 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c6837d12c44b4daf550892e00daa1d" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b" is_pre_election: true
I20260927 07:07:14.135830 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c6837d12c44b4daf550892e00daa1d" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" is_pre_election: true
I20260927 07:07:14.136067 6067 raft_consensus.cc:2454] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:14.136071 6347 raft_consensus.cc:2454] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:14.136592 6136 leader_election.cc:302] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e; no voters:
I20260927 07:07:14.136890 6484 raft_consensus.cc:2781] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:14.137058 6484 raft_consensus.cc:482] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:14.137202 6484 raft_consensus.cc:3037] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.139822 6484 raft_consensus.cc:504] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.140435 6484 leader_election.cc:288] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Requested vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:14.140976 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c6837d12c44b4daf550892e00daa1d" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
I20260927 07:07:14.141055 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c6837d12c44b4daf550892e00daa1d" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b"
I20260927 07:07:14.141211 6067 raft_consensus.cc:3037] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.141317 6347 raft_consensus.cc:3037] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.144277 6067 raft_consensus.cc:2454] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:14.144881 6136 leader_election.cc:302] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e; no voters:
I20260927 07:07:14.145217 6484 raft_consensus.cc:2781] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:14.145246 6347 raft_consensus.cc:2454] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:14.145628 6484 raft_consensus.cc:686] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Becoming Leader. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Running, Role: LEADER
I20260927 07:07:14.146090 6484 consensus_queue.cc:237] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.149051 5905 catalog_manager.cc:5495] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "352598c644d2454d8d9b2bcd7b30f821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:14.226253 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.221217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.30.16.66:42555)
0927 07:07:14.221730 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:14.221734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:14.221750 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:14.221807 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:14.221810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:14.221870 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:14.221953 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:14.222316 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.223155 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.224613 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.225027 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.225121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:14.225576 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:14.225668 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:14.225734 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:14.225933 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 07:07:14.225992 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":276,"threads_started":1}
I20260927 07:07:14.229314 6067 raft_consensus.cc:1261] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:14.229347 6347 raft_consensus.cc:1261] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:14.229940 6484 consensus_queue.cc:1035] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:14.230260 6425 consensus_queue.cc:1035] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:14.245616 5905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:14.245824 5905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:14.248423 5905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Sending DeleteTablet for 3 replicas of tablet 93c6837d12c44b4daf550892e00daa1d
I20260927 07:07:14.249292 6322 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c6837d12c44b4daf550892e00daa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55944
I20260927 07:07:14.249291 6042 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c6837d12c44b4daf550892e00daa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56112
I20260927 07:07:14.249697 6183 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c6837d12c44b4daf550892e00daa1d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:36304
I20260927 07:07:14.250452 5905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:14.251271 5905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:14.252719 6521 tablet_replica.cc:330] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: stopping tablet replica
I20260927 07:07:14.252970 6521 raft_consensus.cc:2229] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:14.253275 6521 raft_consensus.cc:2258] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.254386 6521 ts_tablet_manager.cc:1905] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.262283 6522 tablet_replica.cc:330] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: stopping tablet replica
I20260927 07:07:14.262925 6523 tablet_replica.cc:330] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: stopping tablet replica
I20260927 07:07:14.263211 6042 tablet_service.cc:1467] Processing CreateTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 (DEFAULT_TABLE table=test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:14.263279 6321 tablet_service.cc:1467] Processing CreateTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 (DEFAULT_TABLE table=test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:14.263660 6042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b50c74c3d749c99fa27f13a1f70901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:14.263317 6522 raft_consensus.cc:2229] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:14.263729 6521 ts_tablet_manager.cc:1918] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:14.263795 6321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b50c74c3d749c99fa27f13a1f70901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:14.263893 6521 log.cc:1196] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/93c6837d12c44b4daf550892e00daa1d
I20260927 07:07:14.264210 6522 raft_consensus.cc:2258] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.263643 6523 raft_consensus.cc:2229] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:14.264374 6521 ts_tablet_manager.cc:1939] T 93c6837d12c44b4daf550892e00daa1d P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting consensus metadata
I20260927 07:07:14.264753 6523 raft_consensus.cc:2258] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.265666 5893 catalog_manager.cc:4841] TS 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759): tablet 93c6837d12c44b4daf550892e00daa1d (table test-table [id=e16681a6b6754222a3e33a5fee6eccb4]) successfully deleted
I20260927 07:07:14.266388 6522 ts_tablet_manager.cc:1905] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.267282 6523 ts_tablet_manager.cc:1905] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.271327 6502 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Bootstrap starting.
I20260927 07:07:14.273180 6502 tablet_bootstrap.cc:654] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:14.273381 6522 ts_tablet_manager.cc:1918] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:14.273533 6522 log.cc:1196] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/93c6837d12c44b4daf550892e00daa1d
I20260927 07:07:14.274030 6522 ts_tablet_manager.cc:1939] T 93c6837d12c44b4daf550892e00daa1d P e6ef9684930f4142af5e78fa2626635b: Deleting consensus metadata
I20260927 07:07:14.274675 6523 ts_tablet_manager.cc:1918] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:14.274971 6523 log.cc:1196] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/93c6837d12c44b4daf550892e00daa1d
I20260927 07:07:14.275352 5890 catalog_manager.cc:4841] TS e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283): tablet 93c6837d12c44b4daf550892e00daa1d (table test-table [id=e16681a6b6754222a3e33a5fee6eccb4]) successfully deleted
I20260927 07:07:14.275573 6523 ts_tablet_manager.cc:1939] T 93c6837d12c44b4daf550892e00daa1d P 352598c644d2454d8d9b2bcd7b30f821: Deleting consensus metadata
I20260927 07:07:14.275887 6504 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Bootstrap starting.
I20260927 07:07:14.276194 6183 tablet_service.cc:1467] Processing CreateTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 (DEFAULT_TABLE table=test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:14.276626 6183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b50c74c3d749c99fa27f13a1f70901. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:14.277554 6504 tablet_bootstrap.cc:654] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:14.278789 5890 catalog_manager.cc:4841] TS 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555): tablet 93c6837d12c44b4daf550892e00daa1d (table test-table [id=e16681a6b6754222a3e33a5fee6eccb4]) successfully deleted
I20260927 07:07:14.280622 6504 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: No bootstrap required, opened a new log
I20260927 07:07:14.280788 6504 ts_tablet_manager.cc:1397] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 07:07:14.281526 6504 raft_consensus.cc:348] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.281786 6504 raft_consensus.cc:374] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:14.281872 6504 raft_consensus.cc:729] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9971348bc93b4bff88f8b74f83f9ae1e, State: Initialized, Role: FOLLOWER
I20260927 07:07:14.282116 6504 consensus_queue.cc:260] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.282817 6504 ts_tablet_manager.cc:1428] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 07:07:14.283190 6505 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Bootstrap starting.
I20260927 07:07:14.284487 6502 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: No bootstrap required, opened a new log
I20260927 07:07:14.284689 6502 ts_tablet_manager.cc:1397] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260927 07:07:14.284894 6505 tablet_bootstrap.cc:654] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:14.285696 6502 raft_consensus.cc:348] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.285992 6502 raft_consensus.cc:374] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:14.286082 6502 raft_consensus.cc:729] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ef9684930f4142af5e78fa2626635b, State: Initialized, Role: FOLLOWER
I20260927 07:07:14.286373 6502 consensus_queue.cc:260] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.286947 6505 tablet_bootstrap.cc:492] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: No bootstrap required, opened a new log
I20260927 07:07:14.287127 6505 ts_tablet_manager.cc:1397] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260927 07:07:14.287551 6502 ts_tablet_manager.cc:1428] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:14.287982 6505 raft_consensus.cc:348] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.288205 6505 raft_consensus.cc:374] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:14.288316 6505 raft_consensus.cc:729] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Initialized, Role: FOLLOWER
I20260927 07:07:14.288544 6505 consensus_queue.cc:260] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.289088 6505 ts_tablet_manager.cc:1428] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 07:07:14.428272 6518 raft_consensus.cc:482] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:14.428484 6518 raft_consensus.cc:504] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.429198 6518 leader_election.cc:288] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:14.429869 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b50c74c3d749c99fa27f13a1f70901" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b" is_pre_election: true
I20260927 07:07:14.429869 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b50c74c3d749c99fa27f13a1f70901" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" is_pre_election: true
I20260927 07:07:14.430124 6347 raft_consensus.cc:2454] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:14.430141 6067 raft_consensus.cc:2454] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 0.
I20260927 07:07:14.430678 6136 leader_election.cc:302] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e, e6ef9684930f4142af5e78fa2626635b; no voters:
I20260927 07:07:14.431021 6518 raft_consensus.cc:2781] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:14.431144 6518 raft_consensus.cc:482] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:14.431238 6518 raft_consensus.cc:3037] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.433223 6518 raft_consensus.cc:504] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.433816 6518 leader_election.cc:288] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Requested vote from peers 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759), e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283)
I20260927 07:07:14.434322 6067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b50c74c3d749c99fa27f13a1f70901" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9971348bc93b4bff88f8b74f83f9ae1e"
I20260927 07:07:14.434530 6067 raft_consensus.cc:3037] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.434474 6347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b50c74c3d749c99fa27f13a1f70901" candidate_uuid: "352598c644d2454d8d9b2bcd7b30f821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ef9684930f4142af5e78fa2626635b"
I20260927 07:07:14.434758 6347 raft_consensus.cc:3037] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:14.436594 6067 raft_consensus.cc:2454] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:14.436739 6347 raft_consensus.cc:2454] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 352598c644d2454d8d9b2bcd7b30f821 in term 1.
I20260927 07:07:14.437026 6136 leader_election.cc:302] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 352598c644d2454d8d9b2bcd7b30f821, 9971348bc93b4bff88f8b74f83f9ae1e; no voters:
I20260927 07:07:14.437304 6518 raft_consensus.cc:2781] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:14.437463 6518 raft_consensus.cc:686] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Becoming Leader. State: Replica: 352598c644d2454d8d9b2bcd7b30f821, State: Running, Role: LEADER
I20260927 07:07:14.437740 6518 consensus_queue.cc:237] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } }
I20260927 07:07:14.440315 5906 catalog_manager.cc:5495] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 reported cstate change: term changed from 0 to 1, leader changed from <none> to 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "352598c644d2454d8d9b2bcd7b30f821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "352598c644d2454d8d9b2bcd7b30f821" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:14.484616 6067 raft_consensus.cc:1261] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:14.484731 6347 raft_consensus.cc:1261] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Refusing update from remote peer 352598c644d2454d8d9b2bcd7b30f821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:14.485152 6518 consensus_queue.cc:1035] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9971348bc93b4bff88f8b74f83f9ae1e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:14.485428 6519 consensus_queue.cc:1035] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ef9684930f4142af5e78fa2626635b" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 41283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:14.534904 5904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:14.535147 5904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:14.539144 5904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16a01a472f754ee392436f1c902cdb24: Sending DeleteTablet for 3 replicas of tablet 22b50c74c3d749c99fa27f13a1f70901
I20260927 07:07:14.540174 6042 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56112
I20260927 07:07:14.540653 6521 tablet_replica.cc:330] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: stopping tablet replica
I20260927 07:07:14.541271 6183 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:36304
I20260927 07:07:14.541323 6321 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b50c74c3d749c99fa27f13a1f70901 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:14 UTC) from {username='oryx', principal='oryx/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55944
I20260927 07:07:14.541549 6523 tablet_replica.cc:330] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: stopping tablet replica
I20260927 07:07:14.541647 6522 tablet_replica.cc:330] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: stopping tablet replica
I20260927 07:07:14.541759 6523 raft_consensus.cc:2229] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:14.541857 6522 raft_consensus.cc:2229] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:14.542049 6522 raft_consensus.cc:2258] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.542098 6523 raft_consensus.cc:2258] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.541011 6521 raft_consensus.cc:2229] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:14.542712 6521 raft_consensus.cc:2258] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:14.543205 6522 ts_tablet_manager.cc:1905] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.543740 6523 ts_tablet_manager.cc:1905] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.544672 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5977
I20260927 07:07:14.545938 6521 ts_tablet_manager.cc:1905] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:14.549857 6522 ts_tablet_manager.cc:1918] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:14.550011 6522 log.cc:1196] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/22b50c74c3d749c99fa27f13a1f70901
I20260927 07:07:14.550422 6522 ts_tablet_manager.cc:1939] T 22b50c74c3d749c99fa27f13a1f70901 P e6ef9684930f4142af5e78fa2626635b: Deleting consensus metadata
I20260927 07:07:14.550952 6523 ts_tablet_manager.cc:1918] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:14.551131 6523 log.cc:1196] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/22b50c74c3d749c99fa27f13a1f70901
I20260927 07:07:14.551636 6523 ts_tablet_manager.cc:1939] T 22b50c74c3d749c99fa27f13a1f70901 P 352598c644d2454d8d9b2bcd7b30f821: Deleting consensus metadata
I20260927 07:07:14.551627 5890 catalog_manager.cc:4841] TS e6ef9684930f4142af5e78fa2626635b (127.30.16.67:41283): tablet 22b50c74c3d749c99fa27f13a1f70901 (table test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]) successfully deleted
I20260927 07:07:14.552628 6521 ts_tablet_manager.cc:1918] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 07:07:14.552834 6521 log.cc:1196] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/22b50c74c3d749c99fa27f13a1f70901
I20260927 07:07:14.552975 5890 catalog_manager.cc:4841] TS 352598c644d2454d8d9b2bcd7b30f821 (127.30.16.66:42555): tablet 22b50c74c3d749c99fa27f13a1f70901 (table test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]) successfully deleted
I20260927 07:07:14.553288 6521 ts_tablet_manager.cc:1939] T 22b50c74c3d749c99fa27f13a1f70901 P 9971348bc93b4bff88f8b74f83f9ae1e: Deleting consensus metadata
I20260927 07:07:14.554162 5893 catalog_manager.cc:4841] TS 9971348bc93b4bff88f8b74f83f9ae1e (127.30.16.65:36759): tablet 22b50c74c3d749c99fa27f13a1f70901 (table test-table [id=6012cb0cf5f44e71b6119b0dd0a9c290]) successfully deleted
Sep 27 07:07:14 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.65@KRBTEST.COM
I20260927 07:07:14.574127 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.545969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56134 (local address 127.30.16.65:36759)
0927 07:07:14.558943 (+ 12974us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:14.558947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:14.558964 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:07:14.559012 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:14.559016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:14.559088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:14.559179 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:14.560814 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.561685 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.563247 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.563660 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.563755 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:14.568056 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:14.568071 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:14.568087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:14.568120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:14.571259 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.572012 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.572020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.572022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.572153 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.572605 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.572609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.572611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.573025 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:14.573266 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:14.573576 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 07:07:14.573762 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12829,"thread_start_us":12755,"threads_started":1}
I20260927 07:07:14.574898 6108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:14.697232 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5977
I20260927 07:07:14.720351 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6117
Sep 27 07:07:14 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.66@KRBTEST.COM
I20260927 07:07:14.736068 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.721381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36338 (local address 127.30.16.66:42555)
0927 07:07:14.721581 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:14.721586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:14.721606 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:07:14.721824 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:14.721828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:14.721917 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:14.722089 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:14.723475 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.724226 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.725439 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.725860 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.725965 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:14.730212 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:14.730271 (+ 59us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:14.730299 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:14.730347 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:14.733212 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.733900 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.733912 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.733918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.734081 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.734462 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.734469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.734473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.734880 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:14.735204 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:14.735503 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:07:14.735720 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:07:14.736757 6248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:14.882946 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6117
I20260927 07:07:14.906325 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6257
Sep 27 07:07:14 dist-test-slave-8vqf krb5kdc[5861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.30.16.67@KRBTEST.COM
I20260927 07:07:14.921258 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:14.907173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55956 (local address 127.30.16.67:41283)
0927 07:07:14.907659 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:14.907663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:14.907679 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:07:14.907733 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:14.907737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:14.907796 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:14.907896 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:14.909111 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.909900 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.911220 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:14.911588 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:14.911682 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:14.915862 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:14.915885 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:14.915901 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:14.915940 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:14.918798 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.919491 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.919498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.919500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.919640 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:14.919973 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:14.919976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:14.919978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:14.920335 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:14.920554 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:14.920793 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 07:07:14.920995 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":272,"threads_started":1}
I20260927 07:07:14.921996 6387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:15.044401 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6257
I20260927 07:07:15.066256 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 5874
I20260927 07:07:15.067431 5946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:15.183826 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 5874
2026-09-27T07:07:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7197 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:15 dist-test-slave-8vqf krb5kdc[6544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:15 dist-test-slave-8vqf krb5kdc[6544](info): set up 2 sockets
Sep 27 07:07:15 dist-test-slave-8vqf krb5kdc[6544](info): commencing operation
krb5kdc: starting...
W20260927 07:07:16.287637 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:07:16 dist-test-slave-8vqf krb5kdc[6544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:07:16.524096 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:41385
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:41385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:16.856675 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:16.857051 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:16.857209 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:16.864663 6557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:16.864838 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:16.864949 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:16.865057 6557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:16.865139 6557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:16.875475 6557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:41385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:41385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:16.876892 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:16.878275 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:16.890789 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:16.890883 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:16.890913 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:16.891728 6557 server_base.cc:1034] running on GCE node
I20260927 07:07:16.892561 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:16.894064 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:16.895268 6557 hybrid_clock.cc:648] HybridClock initialized: now 1790492836895226 us; error 44 us; skew 500 ppm
I20260927 07:07:16.895638 6557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T07:07:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1779 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:17 dist-test-slave-8vqf krb5kdc[6571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:17 dist-test-slave-8vqf krb5kdc[6571](info): set up 2 sockets
Sep 27 07:07:17 dist-test-slave-8vqf krb5kdc[6571](info): commencing operation
krb5kdc: starting...
W20260927 07:07:18.075889 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.004s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:07:18 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:07:18.310660 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43633
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:18.644688 6584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:18.645068 6584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:18.645226 6584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:18.652560 6584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:18.652716 6584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:18.652829 6584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:18.652930 6584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:18.653023 6584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:18.663116 6584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:18.664546 6584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:18.665901 6584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:18.677970 6589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:18.677970 6592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:18.678658 6590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:18.679111 6584 server_base.cc:1034] running on GCE node
I20260927 07:07:18.680269 6584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:18.681839 6584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:18.683054 6584 hybrid_clock.cc:648] HybridClock initialized: now 1790492838682994 us; error 72 us; skew 500 ppm
I20260927 07:07:18.683377 6584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:18 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:18.699819 6584 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:07:18.701581 6584 webserver.cc:460] Webserver started at http://127.30.16.126:40797/ using document root <none> and password file <none>
I20260927 07:07:18.702316 6584 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:18.702427 6584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:18.702811 6584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:18.706497 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "5286b9e6caa84671923e78d764409d89"
format_stamp: "Formatted at 2026-09-27 07:07:18 on dist-test-slave-8vqf"
server_key: "191f4f38de371c65f0b70848a60f33a9"
server_key_iv: "7bf17e5fbc3d56f16c0cf0dad94c5865"
server_key_version: "encryptionkey@0"
I20260927 07:07:18.707404 6584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "5286b9e6caa84671923e78d764409d89"
format_stamp: "Formatted at 2026-09-27 07:07:18 on dist-test-slave-8vqf"
server_key: "191f4f38de371c65f0b70848a60f33a9"
server_key_iv: "7bf17e5fbc3d56f16c0cf0dad94c5865"
server_key_version: "encryptionkey@0"
I20260927 07:07:18.712975 6584 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:07:18.716681 6599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:18.718178 6584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:07:18.718321 6584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5286b9e6caa84671923e78d764409d89"
format_stamp: "Formatted at 2026-09-27 07:07:18 on dist-test-slave-8vqf"
server_key: "191f4f38de371c65f0b70848a60f33a9"
server_key_iv: "7bf17e5fbc3d56f16c0cf0dad94c5865"
server_key_version: "encryptionkey@0"
I20260927 07:07:18.718490 6584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:18.745581 6584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:18.750247 6584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:18.750496 6584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:18.770383 6584 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43633
I20260927 07:07:18.770385 6650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43633 every 8 connection(s)
I20260927 07:07:18.771924 6584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:18.773221 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6584
I20260927 07:07:18.773432 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:07:18.774114 30785 external_mini_cluster.cc:1428] Setting key 33356512f41d364fda9d22628c251983
I20260927 07:07:18.776408 6651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:18.787644 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Bootstrap starting.
Sep 27 07:07:18 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:18.791648 6651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:18.792768 6651 log.cc:824] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:18.796207 6651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: No bootstrap required, opened a new log
I20260927 07:07:18.797116 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:18.776200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57672 (local address 127.30.16.126:43633)
0927 07:07:18.777034 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:18.777048 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:18.777089 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:07:18.778782 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:18.778805 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:18.779271 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:18.779701 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:18.781227 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:18.782614 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:18.783902 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:18.784634 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:18.784849 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:18.789463 (+ 4614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:18.789472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:18.789497 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:18.789538 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:18.792733 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:18.793661 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:18.793672 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:18.793684 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:18.793891 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:18.794314 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:18.794318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:18.794320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:18.794915 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:18.795133 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:18.795603 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 07:07:18.795960 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":357,"threads_started":1}
I20260927 07:07:18.804080 6651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:18.804469 6651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:18.804570 6651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5286b9e6caa84671923e78d764409d89, State: Initialized, Role: FOLLOWER
I20260927 07:07:18.805234 6651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:18.805485 6651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:18.805645 6651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:18.805848 6651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:18.808079 6651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:18.808586 6651 leader_election.cc:302] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5286b9e6caa84671923e78d764409d89; no voters:
I20260927 07:07:18.809250 6651 leader_election.cc:288] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:18.809374 6656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:18.810146 6656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 LEADER]: Becoming Leader. State: Replica: 5286b9e6caa84671923e78d764409d89, State: Running, Role: LEADER
I20260927 07:07:18.810763 6656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:18.811681 6651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:18.813689 6657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5286b9e6caa84671923e78d764409d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } } }
I20260927 07:07:18.813789 6658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5286b9e6caa84671923e78d764409d89. Latest consensus state: current_term: 1 leader_uuid: "5286b9e6caa84671923e78d764409d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } } }
I20260927 07:07:18.814155 6657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:18.814328 6658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:18.816558 6661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:18.822024 6661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:18.831686 6661 catalog_manager.cc:1347] Generated new cluster ID: a2e872690dae4e008979f379513c3e50
I20260927 07:07:18.831809 6661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:18.859011 6661 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:18.860384 6661 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:18.869385 6661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Generated new TSK 0
I20260927 07:07:18.870469 6661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:07:18.943357 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:19.286820 6677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:19.287158 6677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:19.287281 6677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:19.294797 6677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:19.294910 6677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:19.295025 6677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:19.305195 6677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:19.306658 6677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:19.308081 6677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:19.321856 6683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:19.321943 6682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:19.321947 6685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:19.323185 6677 server_base.cc:1034] running on GCE node
I20260927 07:07:19.323838 6677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:19.324882 6677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:19.326114 6677 hybrid_clock.cc:648] HybridClock initialized: now 1790492839326047 us; error 65 us; skew 500 ppm
I20260927 07:07:19.326447 6677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:19 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492839, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:19.329699 6677 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:07:19.331331 6677 webserver.cc:460] Webserver started at http://127.30.16.65:37793/ using document root <none> and password file <none>
I20260927 07:07:19.332062 6677 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:19.332204 6677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:19.332523 6677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:19.335541 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "175cc3f2c075469aa9884385ab49ab8c"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "25461a33bbaf4bc24bf96d206cff7897"
server_key_iv: "df6641b533e22d23e0cf4e4833036115"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.336334 6677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "175cc3f2c075469aa9884385ab49ab8c"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "25461a33bbaf4bc24bf96d206cff7897"
server_key_iv: "df6641b533e22d23e0cf4e4833036115"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.341583 6677 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:07:19.345188 6692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.346511 6677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:07:19.346738 6677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "175cc3f2c075469aa9884385ab49ab8c"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "25461a33bbaf4bc24bf96d206cff7897"
server_key_iv: "df6641b533e22d23e0cf4e4833036115"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.346982 6677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:19.368317 6677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:19.372752 6677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:19.373066 6677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:19.374039 6677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:19.375756 6677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:19.375923 6677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.376083 6677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:19.376149 6677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.410641 6677 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:32923
I20260927 07:07:19.410779 6804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:32923 every 8 connection(s)
I20260927 07:07:19.412160 6677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:19.419816 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6677
I20260927 07:07:19.420053 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:19.420986 30785 external_mini_cluster.cc:1428] Setting key 0f6c3019918561e861d3470a46d552bd
Sep 27 07:07:19 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492839, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:19.435111 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.414956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:41035 (local address 127.30.16.126:43633)
0927 07:07:19.415464 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:19.415469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:19.416302 (+ 833us) server_negotiation.cc:407] Connection header received
0927 07:07:19.418314 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:19.418319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:19.418411 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:19.418543 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:19.420137 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.420856 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.422887 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.423399 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.423509 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:19.428272 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:19.428278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:19.428281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:19.428316 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:19.431590 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:19.432389 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:19.432397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:19.432404 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:19.432555 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:19.433020 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:19.433028 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:19.433034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:19.433407 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:19.433552 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:19.434350 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 07:07:19.434628 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":260,"threads_started":1}
I20260927 07:07:19.435834 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.415384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:41035)
0927 07:07:19.416147 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:19.416185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:19.418031 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:19.418807 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:19.418816 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:19.419336 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:19.419921 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:19.419967 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.421013 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.421018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:19.422742 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:19.422760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.423629 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.423637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:19.424049 (+ 412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:19.424822 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:19.424867 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:19.427986 (+ 3119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:19.431762 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:19.431770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:19.431785 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:19.432242 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:19.432663 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:19.432666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:19.432668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:19.432902 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:19.433539 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:19.433546 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:19.433940 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 07:07:19.434176 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 07:07:19.434665 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":266,"threads_started":1}
I20260927 07:07:19.437496 6805 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43633
I20260927 07:07:19.437844 6805 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:19.438786 6805 heartbeater.cc:511] Master 127.30.16.126:43633 requested a full tablet report, sending...
I20260927 07:07:19.441478 6616 ts_manager.cc:194] Registered new tserver with Master: 175cc3f2c075469aa9884385ab49ab8c (127.30.16.65:32923)
I20260927 07:07:19.443037 6616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:41035
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:07:19.474347 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:19.812062 6811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:19.812454 6811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:19.812579 6811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:19.819933 6811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:19.820055 6811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:19.820169 6811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:19.830026 6811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:19.831388 6811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:19.832721 6811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:19.846415 6816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:19.846431 6817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:19.846526 6819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:19.847209 6811 server_base.cc:1034] running on GCE node
I20260927 07:07:19.847846 6811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:19.848883 6811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:19.850118 6811 hybrid_clock.cc:648] HybridClock initialized: now 1790492839850071 us; error 47 us; skew 500 ppm
I20260927 07:07:19.850507 6811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:19 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492839, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:19.853730 6811 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:07:19.855383 6811 webserver.cc:460] Webserver started at http://127.30.16.66:46429/ using document root <none> and password file <none>
I20260927 07:07:19.856091 6811 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:19.856232 6811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:19.856541 6811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:19.859499 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "5950e977812a4a4f93b6c71ba12bcbec"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "7fef6a4f302ceb0512ee48d7d0872197"
server_key_iv: "5d6678d67a64b1087c69cb8a9329222e"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.860289 6811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "5950e977812a4a4f93b6c71ba12bcbec"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "7fef6a4f302ceb0512ee48d7d0872197"
server_key_iv: "5d6678d67a64b1087c69cb8a9329222e"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.865579 6811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:07:19.869381 6826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.870921 6811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:07:19.871104 6811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "5950e977812a4a4f93b6c71ba12bcbec"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "7fef6a4f302ceb0512ee48d7d0872197"
server_key_iv: "5d6678d67a64b1087c69cb8a9329222e"
server_key_version: "encryptionkey@0"
I20260927 07:07:19.871302 6811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:19.890048 6811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:19.894413 6811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:19.894771 6811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:19.895781 6811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:19.897181 6811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:19.897256 6811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.897414 6811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:19.897465 6811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:19.931090 6811 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:36237
I20260927 07:07:19.931186 6938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:36237 every 8 connection(s)
I20260927 07:07:19.932462 6811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:19.936182 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6811
I20260927 07:07:19.936376 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:19.936987 30785 external_mini_cluster.cc:1428] Setting key 55c540651a06c12f38c462fdfaad0bbd
Sep 27 07:07:19 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492839, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:19.955221 6807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.935144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:44265 (local address 127.30.16.126:43633)
0927 07:07:19.935335 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:19.935339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:19.936706 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 07:07:19.939072 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:19.939078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:19.939160 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:19.939257 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:19.941436 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.942190 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.944721 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.945231 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.945361 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:19.949235 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:19.949240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:19.949243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:19.949279 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:19.952068 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:19.952841 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:19.952846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:19.952856 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:19.952989 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:19.953435 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:19.953440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:19.953442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:19.953701 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:19.953857 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:19.954611 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 07:07:19.954843 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:07:19.956380 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:19.935649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:44265)
0927 07:07:19.936507 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:19.936568 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:19.938754 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:19.939551 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:19.939565 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:19.940371 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:19.941236 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:19.941263 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.942351 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.942359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:19.944552 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:19.944569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:19.945481 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:19.945487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:19.945836 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:19.946770 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:19.946798 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:19.948975 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:19.952212 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:19.952222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:19.952247 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:19.952707 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:19.953101 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:19.953107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:19.953111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:19.953308 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:19.953831 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:19.953840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:19.954212 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 07:07:19.954498 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 07:07:19.955124 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":319,"threads_started":1}
I20260927 07:07:19.958058 6939 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43633
I20260927 07:07:19.958374 6939 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:19.959164 6939 heartbeater.cc:511] Master 127.30.16.126:43633 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:07:19.960731 6616 ts_manager.cc:194] Registered new tserver with Master: 5950e977812a4a4f93b6c71ba12bcbec (127.30.16.66:36237)
I20260927 07:07:19.961527 6616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:44265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:07:19.990866 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:20.327450 6944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:20.327766 6944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:20.327867 6944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:20.335354 6944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:20.335480 6944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:20.335595 6944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:20.345553 6944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:20.346989 6944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:20.348307 6944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:20.362111 6950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:20.362058 6952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:20.362627 6949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:20.362972 6944 server_base.cc:1034] running on GCE node
I20260927 07:07:20.363547 6944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:20.364504 6944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:20.365711 6944 hybrid_clock.cc:648] HybridClock initialized: now 1790492840365645 us; error 68 us; skew 500 ppm
I20260927 07:07:20.366003 6944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:20 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492840, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:20.369251 6944 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:07:20.370801 6944 webserver.cc:460] Webserver started at http://127.30.16.67:41915/ using document root <none> and password file <none>
I20260927 07:07:20.371531 6944 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:20.371666 6944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:20.371984 6944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:20.375116 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "6f23771c9e4a42d0adb55654c743e1f5"
format_stamp: "Formatted at 2026-09-27 07:07:20 on dist-test-slave-8vqf"
server_key: "6bd8aaf14dacae02e8cf66e4b68484a1"
server_key_iv: "153bb041e0afb644309d2e5fe1ee4cfe"
server_key_version: "encryptionkey@0"
I20260927 07:07:20.375906 6944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "6f23771c9e4a42d0adb55654c743e1f5"
format_stamp: "Formatted at 2026-09-27 07:07:20 on dist-test-slave-8vqf"
server_key: "6bd8aaf14dacae02e8cf66e4b68484a1"
server_key_iv: "153bb041e0afb644309d2e5fe1ee4cfe"
server_key_version: "encryptionkey@0"
I20260927 07:07:20.382333 6944 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260927 07:07:20.385929 6959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:20.387434 6944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:20.387624 6944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "6f23771c9e4a42d0adb55654c743e1f5"
format_stamp: "Formatted at 2026-09-27 07:07:20 on dist-test-slave-8vqf"
server_key: "6bd8aaf14dacae02e8cf66e4b68484a1"
server_key_iv: "153bb041e0afb644309d2e5fe1ee4cfe"
server_key_version: "encryptionkey@0"
I20260927 07:07:20.387845 6944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:20.417639 6944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:20.421954 6944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:20.422454 6944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:20.423628 6944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:20.425024 6944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:20.425129 6944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:20.425279 6944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:20.425367 6944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:20.446784 6805 heartbeater.cc:503] Master 127.30.16.126:43633 was elected leader, sending a full tablet report...
I20260927 07:07:20.460690 6944 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36129
I20260927 07:07:20.460793 7071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36129 every 8 connection(s)
I20260927 07:07:20.462009 6944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:20.463457 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 6944
I20260927 07:07:20.463647 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:20.464128 30785 external_mini_cluster.cc:1428] Setting key 41f280db67868428c2e54cce9caeae8b
Sep 27 07:07:20 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492840, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:20.483431 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.464887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:37067 (local address 127.30.16.126:43633)
0927 07:07:20.465598 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:20.465604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:20.466649 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 07:07:20.468648 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:20.468652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:20.468717 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:20.468813 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:20.470358 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.471235 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.472713 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.473163 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.473259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:20.476918 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:20.476924 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:20.476928 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:20.476962 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:20.480450 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.481238 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.481242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.481244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.481353 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.481752 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.481755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.481757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.482011 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:20.482115 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:20.482998 (+ 883us) server_negotiation.cc:299] Negotiation successful
0927 07:07:20.483164 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":467,"threads_started":1}
I20260927 07:07:20.484529 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.465611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:37067)
0927 07:07:20.466448 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:20.466505 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:20.468380 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:20.469111 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:20.469120 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:20.469618 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:20.470194 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:20.470212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.471358 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.471361 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:20.472587 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:20.472597 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.473348 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.473353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:20.473559 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:20.474190 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:20.474211 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:20.476688 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:20.480590 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:20.480598 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:20.480613 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:20.481132 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:20.481455 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:20.481459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:20.481461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:20.481658 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:20.482147 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:20.482154 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:20.482511 (+ 357us) client_negotiation.cc:769] Sending connection context
0927 07:07:20.482879 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 07:07:20.483286 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":322,"threads_started":1}
I20260927 07:07:20.485857 7072 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43633
I20260927 07:07:20.486124 7072 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:20.486860 7072 heartbeater.cc:511] Master 127.30.16.126:43633 requested a full tablet report, sending...
I20260927 07:07:20.488314 6616 ts_manager.cc:194] Registered new tserver with Master: 6f23771c9e4a42d0adb55654c743e1f5 (127.30.16.67:36129)
I20260927 07:07:20.489202 6616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:37067
I20260927 07:07:20.492326 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
Entry for principal oryx/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab.
WARNING: no policy specified for oryx/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
Entry for principal oryx/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab.
WARNING: no policy specified for oryx/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
Entry for principal oryx/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab.
I20260927 07:07:20.637723 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6677
Sep 27 07:07:20 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:07:20.655383 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.638946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.30.16.65:32923)
0927 07:07:20.639641 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:20.639648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:20.640179 (+ 531us) server_negotiation.cc:407] Connection header received
0927 07:07:20.640324 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:20.640330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:20.640560 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:20.640713 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:20.642144 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.643273 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.644507 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.645061 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.645154 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:20.649340 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:20.649354 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:20.649371 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:20.649409 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:20.652748 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.653444 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.653470 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.653473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.653596 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.653956 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.653959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.653961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.654282 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:20.654525 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:20.654894 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 07:07:20.655085 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":361,"threads_started":1}
I20260927 07:07:20.656714 6800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:20.742797 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6677
I20260927 07:07:20.759511 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6811
Sep 27 07:07:20 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:07:20.775923 7085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.760375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.30.16.66:36237)
0927 07:07:20.760926 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:20.760933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:20.760957 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:07:20.761016 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:20.761022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:20.761235 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:20.761411 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:20.762652 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.763695 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.764972 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.765543 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.765681 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:20.769999 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:20.770012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:20.770028 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:20.770071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:20.773035 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.773816 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.773823 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.773825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.773941 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.774314 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.774316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.774318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.774725 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:20.775075 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:20.775331 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:07:20.775551 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260927 07:07:20.777196 6934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:20.866214 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6811
I20260927 07:07:20.881979 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6944
Sep 27 07:07:20 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:07:20.899219 7087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:20.883236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.30.16.67:36129)
0927 07:07:20.884043 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:20.884051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:20.884087 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:07:20.884148 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:20.884154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:20.884449 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:20.884590 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:20.885827 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.886942 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.888230 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:20.888780 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:20.888886 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:20.893506 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:20.893517 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:20.893532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:20.893571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:20.896519 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.897239 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.897246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.897248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.897369 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:20.897701 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:20.897705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:20.897707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:20.898077 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:20.898296 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:20.898627 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 07:07:20.898822 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":372,"threads_started":1}
I20260927 07:07:20.900705 7067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:20.985224 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6944
I20260927 07:07:21.001159 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 6584
I20260927 07:07:21.002640 6646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:21.106382 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 6584
I20260927 07:07:21.127172 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43633
--webserver_interface=127.30.16.126
--webserver_port=40797
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:21.462736 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:21.463052 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:21.463227 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:21.470801 7090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:21.470920 7090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:21.471038 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:21.471138 7090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:21.471221 7090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:21.481714 7090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=40797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:21.483227 7090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:21.484565 7090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:21.496636 7095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:21.496681 7098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:21.496706 7096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:21.497802 7090 server_base.cc:1034] running on GCE node
I20260927 07:07:21.498824 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:21.500391 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:21.501637 7090 hybrid_clock.cc:648] HybridClock initialized: now 1790492841501571 us; error 59 us; skew 500 ppm
I20260927 07:07:21.502009 7090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:21 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492841, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:21.505249 7090 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:07:21.506953 7090 webserver.cc:460] Webserver started at http://127.30.16.126:40797/ using document root <none> and password file <none>
I20260927 07:07:21.507669 7090 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:21.507824 7090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:21.513850 7090 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 07:07:21.517066 7105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:21.518517 7090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:07:21.518760 7090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5286b9e6caa84671923e78d764409d89"
format_stamp: "Formatted at 2026-09-27 07:07:18 on dist-test-slave-8vqf"
server_key: "191f4f38de371c65f0b70848a60f33a9"
server_key_iv: "7bf17e5fbc3d56f16c0cf0dad94c5865"
server_key_version: "encryptionkey@0"
I20260927 07:07:21.519806 7090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:21.586437 7090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:21.591064 7090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:21.591354 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:21.610257 7090 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43633
I20260927 07:07:21.610234 7156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43633 every 8 connection(s)
I20260927 07:07:21.611917 7090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:21.619114 7157 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:07:21.620708 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7090
I20260927 07:07:21.621533 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:32923
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:43633
--webserver_port=37793
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:07:21.622641 7157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Bootstrap starting.
I20260927 07:07:21.648331 7157 log.cc:824] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:21.659783 7157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:07:21.660426 7157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Bootstrap complete.
I20260927 07:07:21.673543 7157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:21.674757 7157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5286b9e6caa84671923e78d764409d89, State: Initialized, Role: FOLLOWER
I20260927 07:07:21.675500 7157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:21.675791 7157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:21.675908 7157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:21.676116 7157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:07:21.681922 7157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:21.682606 7157 leader_election.cc:302] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5286b9e6caa84671923e78d764409d89; no voters:
I20260927 07:07:21.683241 7157 leader_election.cc:288] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:07:21.683429 7161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:07:21.684614 7161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [term 2 LEADER]: Becoming Leader. State: Replica: 5286b9e6caa84671923e78d764409d89, State: Running, Role: LEADER
I20260927 07:07:21.685232 7161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } }
I20260927 07:07:21.685451 7157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:21.688036 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5286b9e6caa84671923e78d764409d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } } }
I20260927 07:07:21.688736 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:21.688108 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5286b9e6caa84671923e78d764409d89. Latest consensus state: current_term: 2 leader_uuid: "5286b9e6caa84671923e78d764409d89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5286b9e6caa84671923e78d764409d89" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43633 } } }
I20260927 07:07:21.689333 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:21.690685 7166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:21.697187 7166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:21.698721 7166 catalog_manager.cc:1259] Loaded cluster ID: a2e872690dae4e008979f379513c3e50
I20260927 07:07:21.698889 7166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:21.702912 7166 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:21.704331 7166 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5286b9e6caa84671923e78d764409d89: Loaded TSK: 0
I20260927 07:07:21.705922 7166 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:07:21.982089 7159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:21.982374 7159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:21.982527 7159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:21.990311 7159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:21.990429 7159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:21.990643 7159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:22.000623 7159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.65.keytab
--principal=oryx/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:32923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=37793
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:22.002060 7159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:22.003501 7159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:22.018342 7183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.018342 7184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.019246 7186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.019580 7159 server_base.cc:1034] running on GCE node
I20260927 07:07:22.020268 7159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:22.021374 7159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:22.022648 7159 hybrid_clock.cc:648] HybridClock initialized: now 1790492842022570 us; error 63 us; skew 500 ppm
I20260927 07:07:22.022996 7159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:22.026454 7159 init.cc:377] Logged in from keytab as oryx/127.30.16.65@KRBTEST.COM (short username oryx)
I20260927 07:07:22.028223 7159 webserver.cc:460] Webserver started at http://127.30.16.65:37793/ using document root <none> and password file <none>
I20260927 07:07:22.028966 7159 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:22.029145 7159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:22.034957 7159 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:07:22.037878 7193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.039463 7159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:22.039630 7159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "175cc3f2c075469aa9884385ab49ab8c"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "25461a33bbaf4bc24bf96d206cff7897"
server_key_iv: "df6641b533e22d23e0cf4e4833036115"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.040592 7159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:22.059625 7159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:22.064204 7159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:22.064486 7159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:22.065485 7159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:22.066883 7159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:22.067010 7159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.067140 7159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:22.067225 7159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.101979 7159 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:32923
I20260927 07:07:22.102098 7305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:32923 every 8 connection(s)
I20260927 07:07:22.103569 7159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:22.108358 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7159
I20260927 07:07:22.108994 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:36237
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:43633
--webserver_port=46429
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:07:22.458395 7310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:22.458743 7310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:22.458886 7310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:22.466440 7310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:22.466604 7310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:22.466795 7310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:22.476979 7310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.66.keytab
--principal=oryx/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:36237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=46429
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:22.478426 7310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:22.479820 7310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:22.494678 7316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.494750 7317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:22.495602 7319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:22.495728 7310 server_base.cc:1034] running on GCE node
I20260927 07:07:22.496403 7310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:22.497498 7310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:22.498692 7310 hybrid_clock.cc:648] HybridClock initialized: now 1790492842498643 us; error 44 us; skew 500 ppm
I20260927 07:07:22.499019 7310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492842, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:22.502612 7310 init.cc:377] Logged in from keytab as oryx/127.30.16.66@KRBTEST.COM (short username oryx)
I20260927 07:07:22.504310 7310 webserver.cc:460] Webserver started at http://127.30.16.66:46429/ using document root <none> and password file <none>
I20260927 07:07:22.505016 7310 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:22.505170 7310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:22.511477 7310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 07:07:22.515251 7326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.517081 7310 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 07:07:22.517273 7310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "5950e977812a4a4f93b6c71ba12bcbec"
format_stamp: "Formatted at 2026-09-27 07:07:19 on dist-test-slave-8vqf"
server_key: "7fef6a4f302ceb0512ee48d7d0872197"
server_key_iv: "5d6678d67a64b1087c69cb8a9329222e"
server_key_version: "encryptionkey@0"
I20260927 07:07:22.518268 7310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:22.554632 7310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:22.561424 7310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:22.561795 7310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:22.563220 7310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:22.565086 7310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:22.565201 7310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.565316 7310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:22.565408 7310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:22.627462 7310 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:36237
I20260927 07:07:22.627755 7438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:36237 every 8 connection(s)
I20260927 07:07:22.629191 7310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:22.634629 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7310
I20260927 07:07:22.635936 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:36129
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:43633
--webserver_port=41915
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:22 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
W20260927 07:07:22.989918 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:22.990255 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:22.990434 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:22.998142 7443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:22.998312 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:22.998519 7443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:23.009008 7443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/krb5kdc/oryx_127.30.16.67.keytab
--principal=oryx/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:36129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=41915
--tserver_master_addrs=127.30.16.126:43633
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:23.010480 7443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:23.011870 7443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:23.030750 7449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:23.032272 7450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:23.041064 7453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:23.041426 7443 server_base.cc:1034] running on GCE node
I20260927 07:07:23.042174 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:23.043426 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:23.044647 7443 hybrid_clock.cc:648] HybridClock initialized: now 1790492843044552 us; error 85 us; skew 500 ppm
I20260927 07:07:23.045061 7443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492843, etypes {rep=17 tkt=17 ses=17}, oryx/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:23.048463 7443 init.cc:377] Logged in from keytab as oryx/127.30.16.67@KRBTEST.COM (short username oryx)
I20260927 07:07:23.051890 7443 webserver.cc:460] Webserver started at http://127.30.16.67:41915/ using document root <none> and password file <none>
I20260927 07:07:23.052729 7443 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:23.052901 7443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:23.058931 7443 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:07:23.061862 7459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.063426 7443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:23.063604 7443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "6f23771c9e4a42d0adb55654c743e1f5"
format_stamp: "Formatted at 2026-09-27 07:07:20 on dist-test-slave-8vqf"
server_key: "6bd8aaf14dacae02e8cf66e4b68484a1"
server_key_iv: "153bb041e0afb644309d2e5fe1ee4cfe"
server_key_version: "encryptionkey@0"
I20260927 07:07:23.064567 7443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:23.113579 7443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:23.118237 7443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:23.118542 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:23.119711 7443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:23.121310 7443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:23.121426 7443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.121546 7443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:23.121636 7443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:23.166009 7443 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36129
I20260927 07:07:23.166044 7571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36129 every 8 connection(s)
I20260927 07:07:23.167694 7443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:23.172072 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7443
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:23.232638 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.173119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.30.16.126:43633)
0927 07:07:23.208647 (+ 35528us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.208656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.208680 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:07:23.208750 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.208757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.208864 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.211731 (+ 2867us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.212064 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.213634 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.217543 (+ 3909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.218162 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.221695 (+ 3533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.222038 (+ 343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:23.222052 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:23.222078 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:23.222121 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:23.226479 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.227471 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.227482 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.227495 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.227710 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:23.228184 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:23.228192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:23.228199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:23.228878 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:23.230480 (+ 1602us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:23.230801 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 07:07:23.231325 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35332,"thread_start_us":25670,"threads_started":1}
W20260927 07:07:23.629017 7309 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.511s user 0.000s sys 0.013s
I20260927 07:07:23.631019 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.106713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:33577)
0927 07:07:22.107509 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:22.107569 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:22.109402 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:22.110867 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:22.110877 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:22.111681 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:22.112504 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.112524 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.114106 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.114111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.115403 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.115414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.116497 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.116504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.116783 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.117498 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:22.117527 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:23.629886 (+1512359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":338,"threads_started":1}
W20260927 07:07:23.631894 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:22.106614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:33577 (local address 127.30.16.126:43633)
0927 07:07:22.107462 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:22.107478 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:22.107705 (+ 227us) server_negotiation.cc:407] Connection header received
0927 07:07:22.109962 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:22.109976 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:22.110329 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:22.110719 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:22.112719 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.113932 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.115559 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.116248 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.116468 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.631595 (+1515127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:33577: BlockingRecv error: recv got EOF from 127.30.16.65:33577 (error 108)
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":337,"threads_started":1}
W20260927 07:07:23.633319 7306 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43633 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.16.126:43633: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:23.644058 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.634040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:46265)
0927 07:07:23.634292 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.634310 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.634462 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.634826 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.634830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.635249 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:23.635406 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.635415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.636484 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.636491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.637458 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.637470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.638166 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.638171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.638769 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.639813 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:23.639837 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:23.643738 (+ 3901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260927 07:07:23.644184 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:23.634085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:46265 (local address 127.30.16.126:43633)
0927 07:07:23.634243 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.634250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.634395 (+ 145us) server_negotiation.cc:407] Connection header received
0927 07:07:23.634616 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.634621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.634689 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.634796 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.635550 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.636330 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.637596 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.637990 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.638125 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.643933 (+ 5808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:46265: BlockingRecv error: recv got EOF from 127.30.16.65:46265 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:23 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:23.654163 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.645000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:50895)
0927 07:07:23.645234 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.645254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.645352 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.645676 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.645680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.646001 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:23.646146 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.646154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.647251 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.647256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.648204 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.648215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.648918 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.648924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.649095 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.649675 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:23.649692 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:23.653874 (+ 4182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128}
W20260927 07:07:23.654570 7306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:07:23.654752 7308 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:23.645192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:50895 (local address 127.30.16.126:43633)
0927 07:07:23.645338 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.645344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.645361 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:07:23.645461 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.645466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.645546 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.645644 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.646277 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.647110 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.648342 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.648756 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.648879 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.654051 (+ 5172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:50895: BlockingRecv error: recv got EOF from 127.30.16.65:50895 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 07:07:24.065490 7442 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.422s user 0.010s sys 0.007s
W20260927 07:07:24.067390 7441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:22.633106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:44299 (local address 127.30.16.126:43633)
0927 07:07:22.633811 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:22.633822 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:22.637164 (+ 3342us) server_negotiation.cc:407] Connection header received
0927 07:07:22.637231 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:22.637236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:22.637326 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:22.637462 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:22.639278 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.640113 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.641770 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.642232 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.642403 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.067073 (+1424670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:44299: BlockingRecv error: recv got EOF from 127.30.16.66:44299 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
I20260927 07:07:24.068831 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:22.633728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:44299)
0927 07:07:22.634659 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:22.634715 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:22.636807 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:22.637748 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:22.637757 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:22.638321 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:22.639070 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.639089 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.640283 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.640292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.641594 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:22.641607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:22.642493 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:22.642503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:22.642862 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:22.643643 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:22.643669 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.066950 (+1423281us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":385,"threads_started":1}
W20260927 07:07:24.069883 7439 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43633 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.16.126:43633: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:24.085471 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:24.070552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:57471)
0927 07:07:24.070739 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:24.070761 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:24.070893 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:24.071199 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:24.071203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:24.071579 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:24.071739 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.071747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.072762 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.072768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.073731 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.073741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.074445 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.074452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.074635 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.081612 (+ 6977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:24.081631 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.085162 (+ 3531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 07:07:24.085631 7441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:24.070721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:57471 (local address 127.30.16.126:43633)
0927 07:07:24.070913 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:24.070918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:24.070938 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:07:24.070999 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:24.071004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:24.071072 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:24.071171 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:24.071883 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.072623 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.073870 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.074271 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.074408 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.085376 (+ 10968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:57471: BlockingRecv error: recv got EOF from 127.30.16.66:57471 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:24.097375 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:24.086239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:45415)
0927 07:07:24.086385 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:24.086404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:24.086532 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:24.087340 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:24.087344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:24.087638 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:24.087796 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.087804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.090013 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.090018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.091296 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.091313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.092897 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.092902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.093026 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.093646 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:24.093663 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.097171 (+ 3508us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 07:07:24.097779 7439 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:07:24.098050 7441 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:24.086593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:45415 (local address 127.30.16.126:43633)
0927 07:07:24.086842 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:24.086852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:24.086892 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:07:24.086978 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:24.086987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:24.087134 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:24.087289 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:24.087968 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.089809 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.091530 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.092701 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.092923 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.097585 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:45415: BlockingRecv error: recv got EOF from 127.30.16.66:45415 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 07:07:24.488319 7575 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.271s user 0.003s sys 0.015s
W20260927 07:07:24.489485 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:23.171439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45105 (local address 127.30.16.126:43633)
0927 07:07:23.199191 (+ 27752us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:23.199200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:23.199229 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:07:23.199316 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:23.199324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:23.199434 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:23.199585 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:23.203057 (+ 3472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.204700 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.213843 (+ 9143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.214673 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.214822 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.489244 (+1274422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:45105: BlockingRecv error: recv got EOF from 127.30.16.67:45105 (error 108)
Metrics: {"server-negotiator.queue_time_us":27562,"thread_start_us":36829,"threads_started":1}
I20260927 07:07:24.490026 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:23.172160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:45105)
0927 07:07:23.189318 (+ 17158us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:23.189390 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:23.192595 (+ 3205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:23.200320 (+ 7725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:23.200340 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:23.201479 (+ 1139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:23.202705 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.202743 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.204956 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.204964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.207920 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:23.207954 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:23.215970 (+ 8016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:23.215979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:23.216312 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:23.217180 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:23.217215 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.489034 (+1271819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16819,"thread_start_us":16566,"threads_started":1}
W20260927 07:07:24.490988 7572 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43633 (0 consecutive failures): Not authorized: Failed to ping master at 127.30.16.126:43633: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:24.500919 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:24.491747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:55107)
0927 07:07:24.491962 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:24.491980 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:24.492078 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:24.492519 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:24.492524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:24.492877 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:24.493122 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.493134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.494477 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.494482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.495587 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.495598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.496337 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.496341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.496458 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.497088 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:24.497105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.500687 (+ 3582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 07:07:24.501200 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:24.491811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:55107 (local address 127.30.16.126:43633)
0927 07:07:24.492057 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:24.492061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:24.492080 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:07:24.492238 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:24.492243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:24.492339 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:24.492440 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:24.493287 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.494339 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.495712 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.496181 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.496286 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.500912 (+ 4626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:55107: BlockingRecv error: recv got EOF from 127.30.16.67:55107 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:24.510334 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:24.501630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:33831)
0927 07:07:24.501827 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:24.501844 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:24.501931 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:24.502272 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:24.502275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:24.502655 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:24.502808 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.502815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.503794 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.503798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.504859 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.504870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.505660 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.505666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.505858 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.506575 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:24.506600 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.510041 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 07:07:24.510578 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:24.501761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:33831 (local address 127.30.16.126:43633)
0927 07:07:24.502026 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:24.502030 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:24.502045 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:07:24.502090 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:24.502093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:24.502147 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:24.502227 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:24.502962 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.503663 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.504981 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.505507 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.505630 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.510229 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:33831: BlockingRecv error: recv got EOF from 127.30.16.67:33831 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
W20260927 07:07:24.510699 7572 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:24 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:24.664229 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:24.655264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:47237)
0927 07:07:24.655800 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:24.655826 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:24.655937 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:24.656383 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:24.656388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:24.656804 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:24.656946 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.656954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.657939 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.657942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.658928 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:24.658939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.659590 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.659594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:24.659711 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.660277 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:24.660293 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:24.663957 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
W20260927 07:07:24.664355 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:24.655305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:47237 (local address 127.30.16.126:43633)
0927 07:07:24.655525 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:24.655534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:24.655936 (+ 402us) server_negotiation.cc:407] Connection header received
0927 07:07:24.656101 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:24.656107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:24.656218 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:24.656326 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:24.657089 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.657811 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.659049 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:24.659437 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:24.659527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:24.664165 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:47237: BlockingRecv error: recv got EOF from 127.30.16.65:47237 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:25.109175 7580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:25.098441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:53961)
0927 07:07:25.099043 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:25.099071 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:25.099207 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:25.099870 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:25.099875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:25.100427 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:25.100598 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.100607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.102070 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.102076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.103638 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.103652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.104470 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.104476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.104668 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.105384 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:25.105402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:25.108936 (+ 3534us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":385,"threads_started":1}
W20260927 07:07:25.109475 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:25.098522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:53961 (local address 127.30.16.126:43633)
0927 07:07:25.098770 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:25.098774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:25.099317 (+ 543us) server_negotiation.cc:407] Connection header received
0927 07:07:25.099400 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:25.099405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:25.099494 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:25.099612 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:25.100759 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.101922 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.103788 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.104279 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.104390 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.109173 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:53961: BlockingRecv error: recv got EOF from 127.30.16.66:53961 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:25.520328 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:25.511339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:34213)
0927 07:07:25.511844 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:25.511859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:25.511972 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:25.512329 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:25.512332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:25.512712 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:25.512855 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.512862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.513823 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.513826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.514883 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.514895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.515620 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.515625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.515738 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.516308 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:25.516324 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:25.519988 (+ 3664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":273,"threads_started":1}
W20260927 07:07:25.520483 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:25.511517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:34213 (local address 127.30.16.126:43633)
0927 07:07:25.511741 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:25.511747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:25.511979 (+ 232us) server_negotiation.cc:407] Connection header received
0927 07:07:25.512123 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:25.512127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:25.512202 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:25.512287 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:25.512970 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.513701 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.515015 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.515466 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.515559 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.520221 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:34213: BlockingRecv error: recv got EOF from 127.30.16.67:34213 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:25 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:25.675053 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:25.665273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:60215)
0927 07:07:25.665817 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:25.665834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:25.665946 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:25.666315 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:25.666318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:25.666732 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:25.666902 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.666910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.667910 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.667913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.668972 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:25.668985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.669807 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.669811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:25.669945 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.670603 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:25.670625 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:25.674695 (+ 4070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":328,"threads_started":1}
W20260927 07:07:25.675074 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:25.665331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60215 (local address 127.30.16.126:43633)
0927 07:07:25.665531 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:25.665535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:25.665913 (+ 378us) server_negotiation.cc:407] Connection header received
0927 07:07:25.666085 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:25.666089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:25.666153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:25.666274 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:25.667037 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.667788 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.669157 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:25.669668 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:25.669772 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:25.674871 (+ 5099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:60215: BlockingRecv error: recv got EOF from 127.30.16.65:60215 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:26.119285 7583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:26.110150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:41687)
0927 07:07:26.110679 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:26.110697 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:26.110854 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:26.111217 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:26.111224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:26.111644 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:26.111824 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.111833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.112821 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.112828 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.113760 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.113771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.114604 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.114610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.114788 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.115435 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:26.115452 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:26.119043 (+ 3591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":285,"threads_started":1}
W20260927 07:07:26.119475 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:26.110236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:41687 (local address 127.30.16.126:43633)
0927 07:07:26.110483 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:26.110517 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:26.110778 (+ 261us) server_negotiation.cc:407] Connection header received
0927 07:07:26.110998 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:26.111004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:26.111072 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:26.111179 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:26.111977 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.112693 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.113899 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.114409 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.114616 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.119251 (+ 4635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:41687: BlockingRecv error: recv got EOF from 127.30.16.66:41687 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:26.531797 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:26.521362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:56477)
0927 07:07:26.522015 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:26.522032 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:26.522175 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:26.522640 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:26.522645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:26.523249 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:26.523471 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.523481 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.524704 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.524708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.525758 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.525770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.526639 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.526645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.526780 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.527627 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:26.527646 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:26.531554 (+ 3908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":347,"threads_started":1}
W20260927 07:07:26.532002 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:26.521614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:56477 (local address 127.30.16.126:43633)
0927 07:07:26.521850 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:26.521856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:26.522125 (+ 269us) server_negotiation.cc:407] Connection header received
0927 07:07:26.522285 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:26.522289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:26.522363 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:26.522456 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:26.523620 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.524564 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.525988 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.526403 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.526523 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.531748 (+ 5225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:56477: BlockingRecv error: recv got EOF from 127.30.16.67:56477 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:26 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:26.686335 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:26.676264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:48605)
0927 07:07:26.676804 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:26.676820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:26.676996 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:26.677468 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:26.677475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:26.677890 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:26.678064 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.678072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.679319 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.679324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.680367 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:26.680380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.681149 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.681153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:26.681281 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.681860 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:26.681878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:26.686077 (+ 4199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":288,"threads_started":1}
W20260927 07:07:26.686460 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:26.676371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:48605 (local address 127.30.16.126:43633)
0927 07:07:26.676628 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:26.676638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:26.676920 (+ 282us) server_negotiation.cc:407] Connection header received
0927 07:07:26.677180 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:26.677188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:26.677277 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:26.677451 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:26.678235 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.679126 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.680533 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:26.681018 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:26.681118 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:26.686254 (+ 5136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:48605: BlockingRecv error: recv got EOF from 127.30.16.65:48605 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:27.131300 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:27.120460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:49407)
0927 07:07:27.121363 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:27.121436 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:27.121624 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:27.122030 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:27.122037 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:27.122670 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:27.122981 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.122997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.124177 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.124187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.125265 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.125276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.126065 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.126071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.126283 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.126987 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:27.127006 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:27.130954 (+ 3948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":650,"threads_started":1}
W20260927 07:07:27.131497 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:27.120572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49407 (local address 127.30.16.126:43633)
0927 07:07:27.120790 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:27.120803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:27.121609 (+ 806us) server_negotiation.cc:407] Connection header received
0927 07:07:27.121776 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:27.121780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:27.121842 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:27.121951 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:27.123166 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.124032 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.125420 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.125907 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.126051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.131167 (+ 5116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:49407: BlockingRecv error: recv got EOF from 127.30.16.66:49407 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:27.541922 7587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:27.532783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:47233)
0927 07:07:27.533541 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:27.533559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:27.533728 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:27.534093 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:27.534099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:27.534483 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:27.534719 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.534727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.535730 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.535736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.536675 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.536685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.537356 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.537361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.537549 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.538152 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:27.538168 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:27.541578 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":481,"threads_started":1}
W20260927 07:07:27.542023 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:27.532843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:47233 (local address 127.30.16.126:43633)
0927 07:07:27.533107 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:27.533116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:27.533640 (+ 524us) server_negotiation.cc:407] Connection header received
0927 07:07:27.533866 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:27.533872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:27.533950 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:27.534060 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:27.534850 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.535591 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.536812 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.537207 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.537341 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.541799 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:47233: BlockingRecv error: recv got EOF from 127.30.16.67:47233 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:27 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:27.696802 7588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:27.687371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:58441)
0927 07:07:27.687884 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:27.687900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:27.688054 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:27.688428 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:27.688438 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:27.688836 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:27.689010 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.689018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.690041 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.690047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.691086 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:27.691099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.691855 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.691860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:27.692058 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.692739 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:27.692757 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:27.696425 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":263,"threads_started":1}
W20260927 07:07:27.696902 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:27.687448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:58441 (local address 127.30.16.126:43633)
0927 07:07:27.687710 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:27.687719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:27.687988 (+ 269us) server_negotiation.cc:407] Connection header received
0927 07:07:27.688219 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:27.688223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:27.688286 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:27.688385 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:27.689157 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.689886 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.691230 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:27.691709 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:27.691843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:27.696632 (+ 4789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:58441: BlockingRecv error: recv got EOF from 127.30.16.65:58441 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:28.142107 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:28.132393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:43579)
0927 07:07:28.132871 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:28.132947 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:28.133120 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:28.133484 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:28.133488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:28.133892 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:28.134054 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.134062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.135210 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.135216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.136217 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.136228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.136947 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.136954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.137137 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.137751 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:28.137767 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:28.141842 (+ 4075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
W20260927 07:07:28.142225 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:28.132451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43579 (local address 127.30.16.126:43633)
0927 07:07:28.132680 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:28.132689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:28.133043 (+ 354us) server_negotiation.cc:407] Connection header received
0927 07:07:28.133249 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:28.133255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:28.133325 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:28.133450 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:28.134227 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.135074 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.136357 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.136791 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.136925 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.142046 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:43579: BlockingRecv error: recv got EOF from 127.30.16.66:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:28.553146 7590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:28.543100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:43817)
0927 07:07:28.543721 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:28.543744 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:28.543897 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:28.544297 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:28.544302 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:28.544755 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:28.544997 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.545007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.546115 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.546123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.547452 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.547474 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.548389 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.548394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.548584 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.549159 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:28.549178 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:28.552773 (+ 3595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":361,"threads_started":1}
W20260927 07:07:28.553259 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:28.543173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:43817 (local address 127.30.16.126:43633)
0927 07:07:28.543462 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:28.543472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:28.543894 (+ 422us) server_negotiation.cc:407] Connection header received
0927 07:07:28.544022 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:28.544028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:28.544151 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:28.544268 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:28.545155 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.545968 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.547602 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.548229 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.548377 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.552992 (+ 4615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:43817: BlockingRecv error: recv got EOF from 127.30.16.67:43817 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:28 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:28.706955 7591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:28.697849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:44659)
0927 07:07:28.698341 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:28.698358 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:28.698470 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:28.698803 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:28.698806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:28.699297 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:28.699448 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.699458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.700410 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.700414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.701348 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:28.701358 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.702011 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.702015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:28.702145 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.702911 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:28.702937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:28.706533 (+ 3596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
W20260927 07:07:28.707073 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:28.697949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:44659 (local address 127.30.16.126:43633)
0927 07:07:28.698115 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:28.698120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:28.698430 (+ 310us) server_negotiation.cc:407] Connection header received
0927 07:07:28.698614 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:28.698617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:28.698677 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:28.698756 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:28.699572 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.700278 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.701468 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:28.701871 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:28.701961 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:28.706783 (+ 4822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:44659: BlockingRecv error: recv got EOF from 127.30.16.65:44659 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:29.156566 7592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:29.143117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:45343)
0927 07:07:29.143622 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:29.143642 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:29.143751 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:29.144134 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:29.144138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:29.144561 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:29.144704 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.144713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.145798 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.145801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.146862 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.146874 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.147595 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.147599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.147728 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.152757 (+ 5029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:29.152776 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:29.156332 (+ 3556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
W20260927 07:07:29.156804 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:29.143348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:45343 (local address 127.30.16.126:43633)
0927 07:07:29.143539 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:29.143543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:29.143713 (+ 170us) server_negotiation.cc:407] Connection header received
0927 07:07:29.143918 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:29.143923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:29.143999 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:29.144130 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:29.144828 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.145625 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.147010 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.147468 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.147577 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.156548 (+ 8971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:45343: BlockingRecv error: recv got EOF from 127.30.16.66:45343 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:29.563346 7593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:29.554221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:39061)
0927 07:07:29.554814 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:29.554829 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:29.555003 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:29.555385 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:29.555391 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:29.555834 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:29.555979 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.555988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.557007 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.557014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.557949 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.557959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.558705 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.558710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.558895 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.559526 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:29.559543 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:29.562960 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":366,"threads_started":1}
W20260927 07:07:29.563428 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:29.554367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:39061 (local address 127.30.16.126:43633)
0927 07:07:29.554643 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:29.554649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:29.554947 (+ 298us) server_negotiation.cc:407] Connection header received
0927 07:07:29.555142 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:29.555148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:29.555236 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:29.555344 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:29.556127 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.556863 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.558087 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.558509 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.558694 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.563177 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:39061: BlockingRecv error: recv got EOF from 127.30.16.67:39061 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:29 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:29.717219 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:29.708191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:36957)
0927 07:07:29.708783 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:29.708804 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:29.708940 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:29.709335 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:29.709340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:29.709849 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:29.710051 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.710061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.711216 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.711220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.712128 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:29.712138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.712829 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.712833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:29.712947 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.713535 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:29.713551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:29.716979 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":283,"threads_started":1}
W20260927 07:07:29.717592 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:29.708281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:36957 (local address 127.30.16.126:43633)
0927 07:07:29.708575 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:29.708582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:29.708990 (+ 408us) server_negotiation.cc:407] Connection header received
0927 07:07:29.709060 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:29.709065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:29.709142 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:29.709256 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:29.710188 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.711086 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.712271 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:29.712697 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:29.712795 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:29.717304 (+ 4509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:36957: BlockingRecv error: recv got EOF from 127.30.16.65:36957 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:30.167187 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:30.157456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:56073)
0927 07:07:30.157950 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:30.157969 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:30.158118 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:30.158514 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:30.158518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:30.158907 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:30.159080 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.159089 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.160229 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.160234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.161153 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.161164 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.161936 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.161944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.162180 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.162955 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:30.162984 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:30.166955 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":298,"threads_started":1}
W20260927 07:07:30.167416 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:30.157568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:56073 (local address 127.30.16.126:43633)
0927 07:07:30.157835 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:30.157844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:30.158087 (+ 243us) server_negotiation.cc:407] Connection header received
0927 07:07:30.158261 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:30.158266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:30.158349 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:30.158446 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:30.159247 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.160080 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.161303 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.161743 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.161840 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.167195 (+ 5355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:56073: BlockingRecv error: recv got EOF from 127.30.16.66:56073 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:30.573346 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:30.564319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:57293)
0927 07:07:30.564782 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:30.564797 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:30.564951 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:30.565295 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:30.565300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:30.565719 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:30.565878 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.565887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.566990 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.566994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.567908 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.567919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.568618 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.568624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.568823 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.569441 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:30.569458 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:30.572983 (+ 3525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
W20260927 07:07:30.573547 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:30.564379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:57293 (local address 127.30.16.126:43633)
0927 07:07:30.564589 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:30.564598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:30.564881 (+ 283us) server_negotiation.cc:407] Connection header received
0927 07:07:30.565087 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:30.565091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:30.565163 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:30.565264 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:30.566025 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.566839 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.568055 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.568451 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.568585 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.573286 (+ 4701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:57293: BlockingRecv error: recv got EOF from 127.30.16.67:57293 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:30 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:30.727823 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:30.718279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:38027)
0927 07:07:30.719004 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:30.719044 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:30.719187 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:30.719594 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:30.719599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:30.720130 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:30.720317 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.720326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.721406 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.721410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.722516 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:30.722527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.723515 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.723523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:30.723654 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.724251 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:30.724268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:30.727584 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":302,"threads_started":1}
W20260927 07:07:30.728104 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:30.718354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:38027 (local address 127.30.16.126:43633)
0927 07:07:30.718607 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:30.718614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:30.719117 (+ 503us) server_negotiation.cc:407] Connection header received
0927 07:07:30.719330 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:30.719335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:30.719415 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:30.719508 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:30.720488 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.721254 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.722682 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:30.723276 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:30.723447 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:30.727795 (+ 4348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:38027: BlockingRecv error: recv got EOF from 127.30.16.65:38027 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:31.177999 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:31.168448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:41791)
0927 07:07:31.169060 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:31.169082 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:31.169244 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:31.169600 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:31.169605 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:31.170093 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:31.170276 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.170285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.171425 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.171431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.172400 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.172412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.173175 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.173181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.173347 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.174026 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:31.174044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:31.177642 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":288,"threads_started":1}
W20260927 07:07:31.178107 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:31.168576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:41791 (local address 127.30.16.126:43633)
0927 07:07:31.168834 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:31.168853 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:31.169172 (+ 319us) server_negotiation.cc:407] Connection header received
0927 07:07:31.169372 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:31.169379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:31.169454 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:31.169566 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:31.170434 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.171284 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.172543 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.173005 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.173135 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.177830 (+ 4695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:41791: BlockingRecv error: recv got EOF from 127.30.16.66:41791 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:31.583822 7599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:31.574394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:46299)
0927 07:07:31.575088 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:31.575105 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:31.575214 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:31.575542 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:31.575545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:31.575945 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:31.576154 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.576163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.577199 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.577202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.578166 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.578177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.578973 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.578977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.579119 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.579695 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:31.579712 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:31.583569 (+ 3857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":341,"threads_started":1}
W20260927 07:07:31.583981 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:31.574513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:46299 (local address 127.30.16.126:43633)
0927 07:07:31.574733 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:31.574737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:31.575185 (+ 448us) server_negotiation.cc:407] Connection header received
0927 07:07:31.575319 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:31.575323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:31.575401 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:31.575494 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:31.576330 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.577067 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.578301 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.578821 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.578933 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.583727 (+ 4794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:46299: BlockingRecv error: recv got EOF from 127.30.16.67:46299 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:31 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:31.738234 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:31.728834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:47707)
0927 07:07:31.729434 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:31.729450 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:31.729610 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:31.729955 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:31.729960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:31.730371 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:31.730535 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.730592 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.731651 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.731658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.732618 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:31.732629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.733470 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.733475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:31.733628 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.734421 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:31.734444 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:31.737871 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":326,"threads_started":1}
W20260927 07:07:31.738353 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:31.729050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:47707 (local address 127.30.16.126:43633)
0927 07:07:31.729276 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:31.729282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:31.729533 (+ 251us) server_negotiation.cc:407] Connection header received
0927 07:07:31.729748 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:31.729755 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:31.729819 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:31.729929 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:31.730732 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.731517 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.732758 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:31.733256 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:31.733382 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:31.738078 (+ 4696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:47707: BlockingRecv error: recv got EOF from 127.30.16.65:47707 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:32.189031 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:32.179239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:47243)
0927 07:07:32.179901 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:32.179928 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:32.180074 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:32.180458 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:32.180463 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:32.181008 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:32.181248 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.181259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.182467 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.182473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.183484 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.183495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.184282 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.184290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.184474 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.185091 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:32.185108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:32.188678 (+ 3570us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":369,"threads_started":1}
W20260927 07:07:32.189288 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:32.179449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:47243 (local address 127.30.16.126:43633)
0927 07:07:32.179692 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:32.179699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:32.179998 (+ 299us) server_negotiation.cc:407] Connection header received
0927 07:07:32.180216 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:32.180221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:32.180317 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:32.180422 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:32.181393 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.182315 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.183643 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.184078 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.184216 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.189041 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:47243: BlockingRecv error: recv got EOF from 127.30.16.66:47243 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:32.594458 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:32.585083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:49389)
0927 07:07:32.585585 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:32.585603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:32.585722 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:32.586120 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:32.586124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:32.586653 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:32.586825 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.586834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.588000 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.588004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.588991 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.589002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.589798 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.589804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.589976 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.590775 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:32.590791 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:32.594219 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":281,"threads_started":1}
W20260927 07:07:32.594770 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:32.585151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:49389 (local address 127.30.16.126:43633)
0927 07:07:32.585399 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:32.585406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:32.585691 (+ 285us) server_negotiation.cc:407] Connection header received
0927 07:07:32.585846 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:32.585853 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:32.585940 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:32.586044 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:32.587070 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.587810 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.589145 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.589604 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.589739 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.594427 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:49389: BlockingRecv error: recv got EOF from 127.30.16.67:49389 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:32 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:32.748323 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:32.739203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:42579)
0927 07:07:32.739687 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:32.739704 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:32.739816 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:32.740178 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:32.740182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:32.740617 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:32.740763 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.740772 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.741799 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.741803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.742844 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:32.742857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.743578 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.743582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:32.743718 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.744369 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:32.744386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:32.748084 (+ 3698us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":217,"threads_started":1}
W20260927 07:07:32.748510 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:32.739425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:42579 (local address 127.30.16.126:43633)
0927 07:07:32.739655 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:32.739661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:32.739780 (+ 119us) server_negotiation.cc:407] Connection header received
0927 07:07:32.739918 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:32.739921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:32.739978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:32.740095 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:32.740904 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.741671 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.742988 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:32.743411 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:32.743513 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:32.748296 (+ 4783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:42579: BlockingRecv error: recv got EOF from 127.30.16.65:42579 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:33.199924 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:33.190186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:60889)
0927 07:07:33.190761 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:33.190783 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:33.190889 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:33.191201 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:33.191205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:33.191643 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:33.191788 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.191796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.192925 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.192929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.193994 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.194008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.194787 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.194793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.194930 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.195605 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:33.195622 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:33.199538 (+ 3916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
W20260927 07:07:33.200086 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:33.190308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:60889 (local address 127.30.16.126:43633)
0927 07:07:33.190487 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:33.190492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:33.190917 (+ 425us) server_negotiation.cc:407] Connection header received
0927 07:07:33.190982 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:33.190986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:33.191057 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:33.191141 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:33.191974 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.192791 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.194148 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.194634 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.194746 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.199762 (+ 5016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:60889: BlockingRecv error: recv got EOF from 127.30.16.66:60889 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:33.604837 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:33.595506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:38245)
0927 07:07:33.595993 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:33.596010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:33.596122 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:33.596421 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:33.596424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:33.596842 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:33.597014 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.597023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.597994 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.597998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.599013 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.599025 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.599687 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.599692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.599823 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.600473 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:33.600490 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:33.604552 (+ 4062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
W20260927 07:07:33.604985 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:33.595562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:38245 (local address 127.30.16.126:43633)
0927 07:07:33.595806 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:33.595876 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:33.596113 (+ 237us) server_negotiation.cc:407] Connection header received
0927 07:07:33.596223 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:33.596227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:33.596287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:33.596359 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:33.597151 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.597839 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.599134 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.599532 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.599624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.604767 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:38245: BlockingRecv error: recv got EOF from 127.30.16.67:38245 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:33 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:33.758847 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:33.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:60797)
0927 07:07:33.749880 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:33.749959 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:33.750118 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:33.750554 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:33.750599 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:33.751034 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:33.751226 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.751238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.752412 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.752416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.753434 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:33.753445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.754136 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.754140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:33.754250 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.754980 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:33.754999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:33.758454 (+ 3455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
W20260927 07:07:33.758936 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:33.749533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60797 (local address 127.30.16.126:43633)
0927 07:07:33.749778 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:33.749785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:33.750064 (+ 279us) server_negotiation.cc:407] Connection header received
0927 07:07:33.750270 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:33.750274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:33.750358 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:33.750489 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:33.751407 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.752281 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.753555 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:33.753999 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:33.754100 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:33.758683 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:60797: BlockingRecv error: recv got EOF from 127.30.16.65:60797 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:34.214357 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:34.200876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:43603)
0927 07:07:34.201348 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:34.201368 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:34.201487 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:34.201822 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:34.201825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:34.202197 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:34.202347 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.202356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.203471 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.203475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.204429 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.204440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.205102 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.205106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.205219 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.210481 (+ 5262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:34.210501 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:34.214120 (+ 3619us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":259,"threads_started":1}
W20260927 07:07:34.214725 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:34.201023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43603 (local address 127.30.16.126:43633)
0927 07:07:34.201257 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:34.201263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:34.201469 (+ 206us) server_negotiation.cc:407] Connection header received
0927 07:07:34.201625 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:34.201629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:34.201692 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:34.201782 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:34.202485 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.203327 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.204570 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.204962 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.205058 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.214315 (+ 9257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:43603: BlockingRecv error: recv got EOF from 127.30.16.66:43603 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:34.619839 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:34.605685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:38163)
0927 07:07:34.606201 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:34.606219 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:34.606334 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:34.606714 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:34.606718 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:34.607142 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:34.607311 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.607320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.608495 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.608499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.609523 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.609535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.610226 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.610230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.610342 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.615502 (+ 5160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:34.615530 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:34.619601 (+ 4071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":234,"threads_started":1}
W20260927 07:07:34.620045 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:34.605876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:38163 (local address 127.30.16.126:43633)
0927 07:07:34.606079 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:34.606087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:34.606370 (+ 283us) server_negotiation.cc:407] Connection header received
0927 07:07:34.606429 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:34.606433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:34.606495 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:34.606682 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:34.607512 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.608322 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.609666 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.610086 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.610183 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.619790 (+ 9607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:38163: BlockingRecv error: recv got EOF from 127.30.16.67:38163 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:34 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:34.769886 7609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:34.759877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:52147)
0927 07:07:34.760576 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:34.760600 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:34.760784 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:34.761219 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:34.761224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:34.761773 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:34.761946 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.761955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.763280 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.763286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.764322 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:34.764335 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.765132 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.765136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:34.765254 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.765812 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:34.765828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:34.769668 (+ 3840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":400,"threads_started":1}
W20260927 07:07:34.770072 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:34.759959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:52147 (local address 127.30.16.126:43633)
0927 07:07:34.760247 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:34.760257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:34.760716 (+ 459us) server_negotiation.cc:407] Connection header received
0927 07:07:34.760946 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:34.760954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:34.761051 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:34.761173 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:34.762096 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.763104 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.764469 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:34.764974 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:34.765098 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:34.769857 (+ 4759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:52147: BlockingRecv error: recv got EOF from 127.30.16.65:52147 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:35.225150 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:35.215681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:37065)
0927 07:07:35.216159 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:35.216180 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:35.216296 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:35.216589 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:35.216592 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:35.217009 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:35.217189 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.217198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.218186 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.218190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.219248 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.219273 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.220044 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.220049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.220166 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.220762 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:35.220792 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:35.224819 (+ 4027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
W20260927 07:07:35.225255 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:35.215747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:37065 (local address 127.30.16.126:43633)
0927 07:07:35.215978 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:35.215985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:35.216264 (+ 279us) server_negotiation.cc:407] Connection header received
0927 07:07:35.216394 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:35.216398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:35.216458 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:35.216545 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:35.217311 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.218051 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.219472 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.219910 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.220011 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.224987 (+ 4976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:37065: BlockingRecv error: recv got EOF from 127.30.16.66:37065 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:35.630429 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:35.620814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:36003)
0927 07:07:35.621270 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:35.621287 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:35.621438 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:35.621776 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:35.621780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:35.622202 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:35.622376 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.622384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.623573 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.623577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.624645 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.624665 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.625571 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.625575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.625696 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.626358 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:35.626375 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:35.630131 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":266,"threads_started":1}
W20260927 07:07:35.630615 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:35.620917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:36003 (local address 127.30.16.126:43633)
0927 07:07:35.621159 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:35.621165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:35.621375 (+ 210us) server_negotiation.cc:407] Connection header received
0927 07:07:35.621576 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:35.621579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:35.621638 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:35.621717 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:35.622524 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.623394 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.624789 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.625412 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.625530 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.630342 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:36003: BlockingRecv error: recv got EOF from 127.30.16.67:36003 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:35 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:35.780587 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:35.770962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:38267)
0927 07:07:35.771454 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:35.771473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:35.771615 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:35.771946 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:35.771949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:35.772371 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:35.772551 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.772560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.773602 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.773606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.774629 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:35.774642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.775391 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.775395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:35.775549 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.776213 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:35.776232 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:35.780331 (+ 4099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
W20260927 07:07:35.780747 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:35.771027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:38267 (local address 127.30.16.126:43633)
0927 07:07:35.771278 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:35.771285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:35.771562 (+ 277us) server_negotiation.cc:407] Connection header received
0927 07:07:35.771726 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:35.771730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:35.771790 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:35.771873 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:35.772699 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.773454 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.774756 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:35.775227 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:35.775321 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:35.780540 (+ 5219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:38267: BlockingRecv error: recv got EOF from 127.30.16.65:38267 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:36.235328 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:36.226142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:52239)
0927 07:07:36.226675 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:36.226699 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:36.226858 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:36.227197 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:36.227202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:36.227615 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:36.227779 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.227787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.228779 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.228785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.229757 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.229768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.230485 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.230491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.230855 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.231479 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:36.231496 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:36.235069 (+ 3573us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":300,"threads_started":1}
W20260927 07:07:36.235508 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:36.226215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52239 (local address 127.30.16.126:43633)
0927 07:07:36.226413 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:36.226419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:36.226790 (+ 371us) server_negotiation.cc:407] Connection header received
0927 07:07:36.226983 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:36.226990 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:36.227060 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:36.227159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:36.227925 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.228643 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.229896 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.230329 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.230479 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.235255 (+ 4776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:52239: BlockingRecv error: recv got EOF from 127.30.16.66:52239 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:36.640965 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:36.631450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:54659)
0927 07:07:36.632007 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:36.632026 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:36.632162 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:36.632499 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:36.632502 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:36.633121 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:36.633362 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.633372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.634781 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.634785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.635868 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.635880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.636556 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.636561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.636694 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.637341 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:36.637360 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:36.640728 (+ 3368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":271,"threads_started":1}
W20260927 07:07:36.641248 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:36.631520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:54659 (local address 127.30.16.126:43633)
0927 07:07:36.631753 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:36.631760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:36.632125 (+ 365us) server_negotiation.cc:407] Connection header received
0927 07:07:36.632267 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:36.632270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:36.632344 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:36.632426 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:36.633571 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.634656 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.635997 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.636402 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.636495 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.640929 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:54659: BlockingRecv error: recv got EOF from 127.30.16.67:54659 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:36 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:36.790239 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:36.781570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:56257)
0927 07:07:36.782039 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:36.782065 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:36.782176 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:36.782467 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:36.782470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:36.782928 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:36.783073 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.783082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.784117 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.784121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.785059 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:36.785071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.785841 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.785845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:36.785966 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.786675 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:36.786699 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:36.790005 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":270,"threads_started":1}
W20260927 07:07:36.790509 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:36.781647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:56257 (local address 127.30.16.126:43633)
0927 07:07:36.781826 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:36.781830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:36.782139 (+ 309us) server_negotiation.cc:407] Connection header received
0927 07:07:36.782279 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:36.782283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:36.782342 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:36.782424 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:36.783249 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.783979 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.785189 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:36.785706 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:36.785808 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:36.790244 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:56257: BlockingRecv error: recv got EOF from 127.30.16.65:56257 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.66@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:37.245422 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:37.236489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.66:49061)
0927 07:07:37.237065 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:37.237087 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:37.237251 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:37.237598 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:37.237603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:37.238029 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:37.238174 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.238183 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.239274 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.239280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.240184 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.240194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.240926 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.240932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.241120 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.241737 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:37.241753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:37.245172 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":288,"threads_started":1}
W20260927 07:07:37.245589 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:37.236667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49061 (local address 127.30.16.126:43633)
0927 07:07:37.236888 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:37.236894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:37.237184 (+ 290us) server_negotiation.cc:407] Connection header received
0927 07:07:37.237379 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:37.237385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:37.237456 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:37.237558 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:37.238322 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.239137 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.240329 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.240774 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.240900 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.245349 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.66:49061: BlockingRecv error: recv got EOF from 127.30.16.66:49061 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.67@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:37.651554 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:37.642128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.67:36219)
0927 07:07:37.642645 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:37.642663 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:37.642786 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:37.643174 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:37.643179 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:37.643677 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:37.643899 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.643910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.645046 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.645052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.646013 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.646023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.646804 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.646810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.646989 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.647657 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:37.647674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:37.651302 (+ 3628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":247,"threads_started":1}
W20260927 07:07:37.651742 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:37.642262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:36219 (local address 127.30.16.126:43633)
0927 07:07:37.642498 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:37.642504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:37.642751 (+ 247us) server_negotiation.cc:407] Connection header received
0927 07:07:37.642891 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:37.642894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:37.642977 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:37.643144 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:37.644053 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.644883 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.646152 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.646654 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.646789 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.651499 (+ 4710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.67:36219: BlockingRecv error: recv got EOF from 127.30.16.67:36219 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
Sep 27 07:07:37 dist-test-slave-8vqf krb5kdc[6571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.30.16.65@KRBTEST.COM for oryx/127.30.16.126@KRBTEST.COM, Server not found in Kerberos database
I20260927 07:07:37.800575 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:37.791511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43633 (local address 127.30.16.65:48749)
0927 07:07:37.792091 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:37.792110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:37.792264 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:37.792609 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:37.792615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:37.793104 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:37.793276 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.793285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.794272 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.794277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.795307 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:37.795318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.796013 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.796018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:37.796206 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.796836 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:37.796853 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:37.800323 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.30.16.126:43633: Server oryx/127.30.16.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":270,"threads_started":1}
W20260927 07:07:37.800814 7576 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:37.791708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:48749 (local address 127.30.16.126:43633)
0927 07:07:37.792054 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:37.792061 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:37.792196 (+ 135us) server_negotiation.cc:407] Connection header received
0927 07:07:37.792401 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:37.792406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:37.792475 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:37.792576 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:37.793418 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.794130 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.795445 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:37.795867 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:37.796007 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:37.800538 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.30.16.65:48749: BlockingRecv error: recv got EOF from 127.30.16.65:48749 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260927 07:07:38.174244 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7159
I20260927 07:07:38.187392 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:38.175522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.30.16.65:32923)
0927 07:07:38.176216 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:38.176223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:38.176253 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:07:38.176317 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:38.176324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:38.176611 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:38.176778 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:38.178270 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.179417 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.180645 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.181361 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.181459 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:38.184124 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:38.184132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:38.184147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:38.184185 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:38.186830 (+ 2645us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 07:07:38.187161 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35982: Request ticket server kudu/127.30.16.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":362,"threads_started":1}
W20260927 07:07:38.187516 7620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35982: Request ticket server kudu/127.30.16.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 07:07:38.187525 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7159: Not authorized: Client connection negotiation failed: client connection to 127.30.16.65:32923: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 07:07:38.187760 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7159
I20260927 07:07:38.204923 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7310
W20260927 07:07:38.218686 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7310: Not authorized: Client connection negotiation failed: client connection to 127.30.16.66:36237: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 07:07:38.218781 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:38.206156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.30.16.66:36237)
0927 07:07:38.206970 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:38.206977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:38.207004 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:07:38.207067 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:38.207074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:38.207312 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:38.207532 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:38.208882 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.209903 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.211267 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.211899 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.212172 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:38.215079 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:38.215098 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:38.215113 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:38.215156 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:38.217848 (+ 2692us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 07:07:38.218225 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46648: Request ticket server kudu/127.30.16.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":432,"threads_started":1}
I20260927 07:07:38.218856 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7310
W20260927 07:07:38.218866 7621 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46648: Request ticket server kudu/127.30.16.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 07:07:38.235706 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7443
I20260927 07:07:38.248457 7622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:38.236866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.30.16.67:36129)
0927 07:07:38.237482 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:38.237489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:38.237515 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:07:38.237576 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:38.237583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:38.237837 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:38.238080 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:38.239534 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.240545 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.241756 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.242336 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.242483 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:38.245143 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:38.245151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:38.245166 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:38.245211 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:38.247721 (+ 2510us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 07:07:38.248077 (+ 356us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51748: Request ticket server kudu/127.30.16.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":313,"threads_started":1}
W20260927 07:07:38.248467 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7443: Not authorized: Client connection negotiation failed: client connection to 127.30.16.67:36129: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.30.16.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 07:07:38.248572 7622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51748: Request ticket server kudu/127.30.16.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 07:07:38.248634 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7443
I20260927 07:07:38.264920 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7090
I20260927 07:07:38.266326 7152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:38.382099 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7090
2026-09-27T07:07:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21419 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T07:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:38Z Disabled control of system clock
I20260927 07:07:38.460570 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34883
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 07:07:38.789089 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:38.789387 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:38.789531 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:38.796815 7627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:38.796942 7627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:38.797012 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:38.797097 7627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:38.797156 7627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:38.806490 7627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:34883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:38.807962 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:38.809794 7627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:38.822131 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:38.822188 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:38.822211 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:38.822389 7627 server_base.cc:1034] running on GCE node
I20260927 07:07:38.823843 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:38.825260 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:38.826516 7627 hybrid_clock.cc:648] HybridClock initialized: now 1790492858826444 us; error 43 us; skew 500 ppm
I20260927 07:07:38.826924 7627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:38.829478 7627 webserver.cc:460] Webserver started at http://127.30.16.126:38257/ using document root <none> and password file <none>
I20260927 07:07:38.830217 7627 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:38.830353 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:38.830753 7627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:38.833662 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "a6918fc7a3a54f98b5437b447a794a21"
format_stamp: "Formatted at 2026-09-27 07:07:38 on dist-test-slave-8vqf"
server_key: "bbc77c09015316d83643d6e7983ff82d"
server_key_iv: "14caf00ee224b42e34590bb646f2cb18"
server_key_version: "encryptionkey@0"
I20260927 07:07:38.834442 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "a6918fc7a3a54f98b5437b447a794a21"
format_stamp: "Formatted at 2026-09-27 07:07:38 on dist-test-slave-8vqf"
server_key: "bbc77c09015316d83643d6e7983ff82d"
server_key_iv: "14caf00ee224b42e34590bb646f2cb18"
server_key_version: "encryptionkey@0"
I20260927 07:07:38.839603 7627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:07:38.843171 7641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:38.844628 7627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:07:38.844820 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "a6918fc7a3a54f98b5437b447a794a21"
format_stamp: "Formatted at 2026-09-27 07:07:38 on dist-test-slave-8vqf"
server_key: "bbc77c09015316d83643d6e7983ff82d"
server_key_iv: "14caf00ee224b42e34590bb646f2cb18"
server_key_version: "encryptionkey@0"
I20260927 07:07:38.845039 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:38.864909 7627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:38.865804 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:38.866102 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:38.884392 7627 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:34883
I20260927 07:07:38.884431 7692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:34883 every 8 connection(s)
I20260927 07:07:38.885802 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:38.889868 7693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:38.892418 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7627
I20260927 07:07:38.892616 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:07:38.893245 30785 external_mini_cluster.cc:1428] Setting key 91ed56232b793cf21c69fccdb215d207
I20260927 07:07:38.903420 7693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21: Bootstrap starting.
I20260927 07:07:38.907393 7693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:38.908567 7693 log.cc:824] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:38.909127 7696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:38.895440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.30.16.126:34883)
0927 07:07:38.896530 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:38.896545 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:38.896596 (+ 51us) server_negotiation.cc:407] Connection header received
0927 07:07:38.898875 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:38.898907 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:38.899562 (+ 655us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:38.900023 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:38.900603 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.902645 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.904073 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:38.905106 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:38.905307 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:38.906585 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:38.906599 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:38.906621 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:38.907329 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:38.907426 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:38.907447 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:38.907582 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:38.907780 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 07:07:38.908100 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":338,"threads_started":1}
I20260927 07:07:38.911660 7693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21: No bootstrap required, opened a new log
I20260927 07:07:38.919538 7693 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } }
I20260927 07:07:38.919983 7693 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:38.920099 7693 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6918fc7a3a54f98b5437b447a794a21, State: Initialized, Role: FOLLOWER
I20260927 07:07:38.920672 7693 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } }
I20260927 07:07:38.920912 7693 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:38.921048 7693 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:38.921250 7693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:38.923658 7693 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } }
I20260927 07:07:38.924224 7693 leader_election.cc:302] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6918fc7a3a54f98b5437b447a794a21; no voters:
I20260927 07:07:38.925045 7693 leader_election.cc:288] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:38.925206 7698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:38.926627 7698 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [term 1 LEADER]: Becoming Leader. State: Replica: a6918fc7a3a54f98b5437b447a794a21, State: Running, Role: LEADER
I20260927 07:07:38.927189 7698 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } }
I20260927 07:07:38.927704 7693 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:38.930043 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6918fc7a3a54f98b5437b447a794a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } } }
I20260927 07:07:38.930392 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:38.930809 7700 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6918fc7a3a54f98b5437b447a794a21. Latest consensus state: current_term: 1 leader_uuid: "a6918fc7a3a54f98b5437b447a794a21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6918fc7a3a54f98b5437b447a794a21" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34883 } } }
I20260927 07:07:38.931285 7700 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:38.938102 7705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:38.942638 7705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:38.950997 7705 catalog_manager.cc:1347] Generated new cluster ID: cfd106e86aa248c4931f54a73929ec6b
I20260927 07:07:38.951122 7705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:38.999297 7705 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:39.000638 7705 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:39.014076 7705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6918fc7a3a54f98b5437b447a794a21: Generated new TSK 0
I20260927 07:07:39.015079 7705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:07:39.024448 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--builtin_ntp_servers=127.30.16.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 07:07:39.364338 7717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:39.364647 7717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:39.364787 7717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:39.372553 7717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:39.372673 7717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:39.372833 7717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:39.382050 7717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:39.383467 7717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:39.385281 7717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:39.398669 7723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:39.398705 7725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:39.398813 7722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:39.399281 7717 server_base.cc:1034] running on GCE node
I20260927 07:07:39.399948 7717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:39.400969 7717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:39.402202 7717 hybrid_clock.cc:648] HybridClock initialized: now 1790492859402155 us; error 43 us; skew 500 ppm
I20260927 07:07:39.402628 7717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:39.411877 7717 webserver.cc:460] Webserver started at http://127.30.16.65:43649/ using document root <none> and password file <none>
I20260927 07:07:39.412747 7717 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:39.412844 7717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:39.413172 7717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:39.416189 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "49fd44802660450c90982edeaa42e87e"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "0b6bf1764a3aff9f7e9e150c7e959af3"
server_key_iv: "74bcf7772d78d54cd5542cadc97929f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.417037 7717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "49fd44802660450c90982edeaa42e87e"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "0b6bf1764a3aff9f7e9e150c7e959af3"
server_key_iv: "74bcf7772d78d54cd5542cadc97929f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.422338 7717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:07:39.425915 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.427421 7717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:39.427619 7717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "49fd44802660450c90982edeaa42e87e"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "0b6bf1764a3aff9f7e9e150c7e959af3"
server_key_iv: "74bcf7772d78d54cd5542cadc97929f0"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.427877 7717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:39.444718 7717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:39.445590 7717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:39.445894 7717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:39.446941 7717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:39.448457 7717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:39.448588 7717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.448730 7717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:39.448815 7717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.483238 7717 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:43237
I20260927 07:07:39.483285 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:43237 every 8 connection(s)
I20260927 07:07:39.484591 7717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:39.491462 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7717
I20260927 07:07:39.491663 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:39.492452 30785 external_mini_cluster.cc:1428] Setting key 2141db5c6010d5b554b43f2654bfb0d9
I20260927 07:07:39.495218 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--builtin_ntp_servers=127.30.16.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 07:07:39.498123 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:39.487226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:55445 (local address 127.30.16.126:34883)
0927 07:07:39.487658 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:39.487663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:39.488918 (+ 1255us) server_negotiation.cc:407] Connection header received
0927 07:07:39.490799 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:39.490803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:39.490896 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:39.490993 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:39.492605 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.493555 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.495249 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.495734 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.495872 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:39.497283 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:39.497294 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:39.497297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:39.497316 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:39.497354 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:39.497365 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:39.497461 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:39.497683 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:07:39.497779 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":210,"threads_started":1}
I20260927 07:07:39.499264 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:39.487856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34883 (local address 127.30.16.65:55445)
0927 07:07:39.488754 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:39.488790 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:39.490444 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:39.491320 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:39.491334 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:39.491471 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:39.492361 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:39.492389 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.493719 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.493723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:39.495105 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:39.495117 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.495966 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.495974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:39.496220 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:39.496947 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:39.496949 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:39.496998 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:39.497089 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:39.497459 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:39.497471 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:39.497482 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 07:07:39.497676 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 07:07:39.498105 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":273,"threads_started":1}
I20260927 07:07:39.500883 7844 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34883
I20260927 07:07:39.501214 7844 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:39.501988 7844 heartbeater.cc:511] Master 127.30.16.126:34883 requested a full tablet report, sending...
I20260927 07:07:39.504089 7658 ts_manager.cc:194] Registered new tserver with Master: 49fd44802660450c90982edeaa42e87e (127.30.16.65:43237)
I20260927 07:07:39.505576 7658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.65:55445
W20260927 07:07:39.825541 7848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:39.825904 7848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:39.826078 7848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:39.834039 7848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:39.834185 7848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:39.834357 7848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:39.843652 7848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:39.845045 7848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:39.846882 7848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:39.860522 7854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:39.860562 7853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:39.860710 7856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:39.861441 7848 server_base.cc:1034] running on GCE node
I20260927 07:07:39.862036 7848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:39.863057 7848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:39.864259 7848 hybrid_clock.cc:648] HybridClock initialized: now 1790492859864209 us; error 41 us; skew 500 ppm
I20260927 07:07:39.864601 7848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:39.867134 7848 webserver.cc:460] Webserver started at http://127.30.16.66:32951/ using document root <none> and password file <none>
I20260927 07:07:39.867904 7848 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:39.867993 7848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:39.868359 7848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:39.871395 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "264fd42de2ba4e1aa54ef18781c43eeb"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "5e417c010a6d50fb0a9d872989233db6"
server_key_iv: "9780905a8a2707ced70391406567917c"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.872206 7848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "264fd42de2ba4e1aa54ef18781c43eeb"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "5e417c010a6d50fb0a9d872989233db6"
server_key_iv: "9780905a8a2707ced70391406567917c"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.877424 7848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:07:39.881050 7862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.882699 7848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 07:07:39.882891 7848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "264fd42de2ba4e1aa54ef18781c43eeb"
format_stamp: "Formatted at 2026-09-27 07:07:39 on dist-test-slave-8vqf"
server_key: "5e417c010a6d50fb0a9d872989233db6"
server_key_iv: "9780905a8a2707ced70391406567917c"
server_key_version: "encryptionkey@0"
I20260927 07:07:39.883138 7848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:39.914602 7848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:39.915608 7848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:39.915959 7848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:39.917063 7848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:39.918676 7848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:39.918810 7848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.918982 7848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:39.919082 7848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:39.956899 7848 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:36111
I20260927 07:07:39.956955 7974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:36111 every 8 connection(s)
I20260927 07:07:39.958272 7848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:39.968247 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7848
I20260927 07:07:39.968472 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:39.969007 30785 external_mini_cluster.cc:1428] Setting key 746b562b20477ad120b7ad03a309179c
I20260927 07:07:39.971446 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:39.961092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49065 (local address 127.30.16.126:34883)
0927 07:07:39.961288 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:39.961293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:39.962845 (+ 1552us) server_negotiation.cc:407] Connection header received
0927 07:07:39.964786 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:39.964790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:39.964859 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:39.965008 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:39.966048 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.966823 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.968369 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.968971 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.969103 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:39.970586 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:39.970593 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:39.970599 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:39.970623 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:39.970652 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:39.970664 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:39.970763 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:39.971052 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 07:07:39.971147 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:07:39.971836 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--builtin_ntp_servers=127.30.16.84:33445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 07:07:39.972645 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:39.961629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34883 (local address 127.30.16.66:49065)
0927 07:07:39.962686 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:39.962728 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:39.964502 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:39.965206 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:39.965215 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:39.965315 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:39.965891 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:39.965908 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.966971 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.966975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:39.968220 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:39.968232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:39.969257 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:39.969263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:39.969501 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:39.970232 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:39.970235 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:39.970276 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:39.970360 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:39.970762 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:39.970775 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:39.970785 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:07:39.971054 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 07:07:39.971465 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":290,"threads_started":1}
I20260927 07:07:39.974160 7975 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34883
I20260927 07:07:39.974483 7975 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:39.975433 7975 heartbeater.cc:511] Master 127.30.16.126:34883 requested a full tablet report, sending...
I20260927 07:07:39.977030 7658 ts_manager.cc:194] Registered new tserver with Master: 264fd42de2ba4e1aa54ef18781c43eeb (127.30.16.66:36111)
I20260927 07:07:39.978250 7658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.66:49065
W20260927 07:07:40.306016 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:40.306334 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:40.306423 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:40.314064 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:40.314183 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:40.314291 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:40.323918 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34883
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:40.325219 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:40.327085 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:40.340842 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:40.340799 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:40.340883 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:40.341950 7978 server_base.cc:1034] running on GCE node
I20260927 07:07:40.342690 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:40.343685 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:40.344969 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790492860344852 us; error 36 us; skew 500 ppm
I20260927 07:07:40.345319 7978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:40.347702 7978 webserver.cc:460] Webserver started at http://127.30.16.67:42231/ using document root <none> and password file <none>
I20260927 07:07:40.348471 7978 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:40.348559 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:40.349001 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:40.352085 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "d74f78c67f064087a194a8e1fe838756"
format_stamp: "Formatted at 2026-09-27 07:07:40 on dist-test-slave-8vqf"
server_key: "deacb57d0675132966798b6755dcf413"
server_key_iv: "b619d89776863ef73582b000af5df79e"
server_key_version: "encryptionkey@0"
I20260927 07:07:40.352927 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "d74f78c67f064087a194a8e1fe838756"
format_stamp: "Formatted at 2026-09-27 07:07:40 on dist-test-slave-8vqf"
server_key: "deacb57d0675132966798b6755dcf413"
server_key_iv: "b619d89776863ef73582b000af5df79e"
server_key_version: "encryptionkey@0"
I20260927 07:07:40.358294 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:07:40.361881 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:40.363390 7978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:40.363576 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "d74f78c67f064087a194a8e1fe838756"
format_stamp: "Formatted at 2026-09-27 07:07:40 on dist-test-slave-8vqf"
server_key: "deacb57d0675132966798b6755dcf413"
server_key_iv: "b619d89776863ef73582b000af5df79e"
server_key_version: "encryptionkey@0"
I20260927 07:07:40.363812 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:40.378746 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:40.379691 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:40.379992 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:40.380962 7978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:40.382428 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:40.382553 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:40.382740 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:40.382833 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:40.416556 7978 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:33961
I20260927 07:07:40.416646 8104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:33961 every 8 connection(s)
I20260927 07:07:40.417878 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:40.426066 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 7978
I20260927 07:07:40.426263 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:40.426896 30785 external_mini_cluster.cc:1428] Setting key f4869f572c5f39034c53a14d7ff6de39
I20260927 07:07:40.432598 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:40.420679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:46487 (local address 127.30.16.126:34883)
0927 07:07:40.420886 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:40.420892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:40.422427 (+ 1535us) server_negotiation.cc:407] Connection header received
0927 07:07:40.424377 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:40.424381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:40.424449 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:40.424559 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:40.425722 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.426596 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.429188 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.429846 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.429997 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:40.431611 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:40.431616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:40.431619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:40.431637 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:40.431667 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:40.431678 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:40.431779 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:40.432218 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 07:07:40.432309 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:07:40.433722 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:40.421301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34883 (local address 127.30.16.67:46487)
0927 07:07:40.422261 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:40.422306 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:40.424110 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:40.424778 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:40.424787 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:40.424884 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:40.425513 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:40.425531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.426764 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.426770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:40.429026 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:40.429042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.430118 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.430129 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:40.430477 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:40.431272 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:40.431275 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:40.431335 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:40.431442 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:40.431807 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:40.431820 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:40.431829 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 07:07:40.432030 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 07:07:40.432502 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":285,"threads_started":1}
I20260927 07:07:40.435041 8105 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34883
I20260927 07:07:40.435354 8105 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:40.436161 8105 heartbeater.cc:511] Master 127.30.16.126:34883 requested a full tablet report, sending...
I20260927 07:07:40.437561 7658 ts_manager.cc:194] Registered new tserver with Master: d74f78c67f064087a194a8e1fe838756 (127.30.16.67:33961)
I20260927 07:07:40.438352 7658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.67:46487
I20260927 07:07:40.442713 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:07:40.449800 7846 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:40.447761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.30.16.126:34883)
0927 07:07:40.447970 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:40.447978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:40.448517 (+ 539us) server_negotiation.cc:407] Connection header received
0927 07:07:40.448668 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:40.448672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:40.448743 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:40.448827 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:40.449471 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53444: BlockingRecv error: recv got EOF from 127.0.0.1:53444 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:07:40.450182 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7717
I20260927 07:07:40.458621 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:40.451309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48548 (local address 127.30.16.65:43237)
0927 07:07:40.451860 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:40.451867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:40.451892 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:07:40.451953 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:40.451959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:40.452180 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:40.452305 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:40.452663 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.453652 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.455160 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.455714 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.455812 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:40.457145 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:40.457154 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:40.457168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:40.457740 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:40.457793 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:40.457811 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:40.457898 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:40.458070 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 07:07:40.458218 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":303,"threads_started":1}
I20260927 07:07:40.459986 7839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:40.547847 7844 heartbeater.cc:503] Master 127.30.16.126:34883 was elected leader, sending a full tablet report...
I20260927 07:07:40.549351 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7717
W20260927 07:07:40.567034 7645 connection.cc:535] server connection from 127.30.16.65:55445 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:07:40.567260 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7848
I20260927 07:07:40.576203 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:40.568380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.30.16.66:36111)
0927 07:07:40.568989 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:40.568997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:40.569026 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:07:40.569090 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:40.569096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:40.569334 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:40.569521 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:40.569963 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.571145 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.572499 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.573135 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.573283 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:40.574542 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:40.574644 (+ 102us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:40.574664 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:40.575323 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:40.575373 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:40.575393 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:40.575559 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:40.575666 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 07:07:40.575807 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":311,"threads_started":1}
I20260927 07:07:40.577666 7970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:40.665256 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7848
I20260927 07:07:40.681548 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7978
I20260927 07:07:40.690873 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:40.682632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.30.16.67:33961)
0927 07:07:40.683298 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:40.683306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:40.683342 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:07:40.683409 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:40.683416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:40.683639 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:40.683917 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:40.684338 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.685493 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.687014 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:40.687678 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:40.687823 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:40.689079 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:40.689088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:40.689104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:40.689783 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:40.689845 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:40.689871 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:40.690001 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:40.690245 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 07:07:40.690407 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":348,"threads_started":1}
I20260927 07:07:40.692189 8100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:40.775556 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7978
I20260927 07:07:40.793393 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 7627
I20260927 07:07:40.794785 7688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:40.900852 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 7627
2026-09-27T07:07:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2512 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T07:07:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:40Z Disabled control of system clock
I20260927 07:07:40.973519 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38575
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:07:41.313818 8124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:41.314170 8124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:41.314327 8124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:41.321836 8124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:41.321979 8124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:41.322050 8124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:41.322146 8124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:41.322217 8124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:41.331918 8124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:38575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:41.333330 8124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:41.335378 8124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:41.348443 8129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:41.348388 8132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:41.348420 8130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:41.349320 8124 server_base.cc:1034] running on GCE node
I20260927 07:07:41.350242 8124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:41.351910 8124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:41.353212 8124 hybrid_clock.cc:648] HybridClock initialized: now 1790492861353153 us; error 53 us; skew 500 ppm
I20260927 07:07:41.353645 8124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:41.356181 8124 webserver.cc:460] Webserver started at http://127.30.16.126:34721/ using document root <none> and password file <none>
I20260927 07:07:41.356920 8124 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:41.357054 8124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:41.357381 8124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:41.362252 8124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "c113ebc5e7a64be1a1f5697708910010"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "d0b36ee521b4ff7de3cf63cfc81dfad5"
server_key_iv: "2a85f7dcfe166ef8c631fecf1fd8f168"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.363170 8124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "c113ebc5e7a64be1a1f5697708910010"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "d0b36ee521b4ff7de3cf63cfc81dfad5"
server_key_iv: "2a85f7dcfe166ef8c631fecf1fd8f168"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.368469 8124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:07:41.372242 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:41.373874 8124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:07:41.374091 8124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "c113ebc5e7a64be1a1f5697708910010"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "d0b36ee521b4ff7de3cf63cfc81dfad5"
server_key_iv: "2a85f7dcfe166ef8c631fecf1fd8f168"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.374329 8124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:41.380620 8124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:41.381454 8124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:41.381754 8124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:41.400310 8124 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:38575
I20260927 07:07:41.400341 8189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:38575 every 8 connection(s)
I20260927 07:07:41.401752 8124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:41.405696 8190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:41.406498 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8124
I20260927 07:07:41.406771 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:07:41.407572 30785 external_mini_cluster.cc:1428] Setting key fa9944cf0b9ed557c9e549e5e237d0ff
I20260927 07:07:41.416354 8193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:41.409727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.30.16.126:38575)
0927 07:07:41.410839 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:41.410862 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:41.410903 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:07:41.412475 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:41.412497 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:41.412921 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:41.413332 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:41.414648 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:41.414654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:41.414667 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:41.414681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:41.414740 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:41.414756 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:41.415049 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:41.415189 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:07:41.415435 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":378,"threads_started":1}
I20260927 07:07:41.419270 8190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010: Bootstrap starting.
I20260927 07:07:41.422189 8190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:41.423285 8190 log.cc:824] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:41.426321 8190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010: No bootstrap required, opened a new log
I20260927 07:07:41.433923 8190 raft_consensus.cc:348] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } }
I20260927 07:07:41.434424 8190 raft_consensus.cc:374] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:41.434530 8190 raft_consensus.cc:729] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c113ebc5e7a64be1a1f5697708910010, State: Initialized, Role: FOLLOWER
I20260927 07:07:41.435173 8190 consensus_queue.cc:260] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } }
I20260927 07:07:41.435431 8190 raft_consensus.cc:388] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:41.435573 8190 raft_consensus.cc:482] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:41.435781 8190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:41.438066 8190 raft_consensus.cc:504] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } }
I20260927 07:07:41.438889 8190 leader_election.cc:302] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c113ebc5e7a64be1a1f5697708910010; no voters:
I20260927 07:07:41.439531 8190 leader_election.cc:288] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:41.439704 8195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:41.440506 8195 raft_consensus.cc:686] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [term 1 LEADER]: Becoming Leader. State: Replica: c113ebc5e7a64be1a1f5697708910010, State: Running, Role: LEADER
I20260927 07:07:41.441129 8195 consensus_queue.cc:237] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } }
I20260927 07:07:41.441856 8190 sys_catalog.cc:564] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:41.444739 8197 sys_catalog.cc:455] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c113ebc5e7a64be1a1f5697708910010. Latest consensus state: current_term: 1 leader_uuid: "c113ebc5e7a64be1a1f5697708910010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } } }
I20260927 07:07:41.445204 8197 sys_catalog.cc:458] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:41.445696 8196 sys_catalog.cc:455] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c113ebc5e7a64be1a1f5697708910010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c113ebc5e7a64be1a1f5697708910010" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38575 } } }
I20260927 07:07:41.446161 8196 sys_catalog.cc:458] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010 [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:41.447849 8202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:41.453871 8202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:41.463245 8202 catalog_manager.cc:1347] Generated new cluster ID: 510948ea6f804e82977329d428f33e25
I20260927 07:07:41.463459 8202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:41.483911 8202 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:41.484383 8202 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:41.492274 8202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c113ebc5e7a64be1a1f5697708910010: Generated new TSK 0
I20260927 07:07:41.493340 8202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:07:41.501958 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--builtin_ntp_servers=127.30.16.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 07:07:41.843107 8214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:41.843467 8214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:41.843622 8214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:41.851105 8214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:41.851209 8214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:41.851395 8214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:41.860842 8214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:41.862219 8214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:41.864252 8214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:41.878495 8219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:41.878592 8220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:41.878656 8214 server_base.cc:1034] running on GCE node
W20260927 07:07:41.878748 8222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:41.879817 8214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:41.880795 8214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:41.882054 8214 hybrid_clock.cc:648] HybridClock initialized: now 1790492861882000 us; error 41 us; skew 500 ppm
I20260927 07:07:41.882421 8214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:41.884845 8214 webserver.cc:460] Webserver started at http://127.30.16.65:33347/ using document root <none> and password file <none>
I20260927 07:07:41.885586 8214 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:41.885721 8214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:41.886076 8214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:41.889205 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "b0d481779c1942348e69975f497d73f6"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "8705f1fb21636e301963c9513e1bc206"
server_key_iv: "401b1e38ada272b8c67920d4996ae9e1"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.890031 8214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "b0d481779c1942348e69975f497d73f6"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "8705f1fb21636e301963c9513e1bc206"
server_key_iv: "401b1e38ada272b8c67920d4996ae9e1"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.895536 8214 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 07:07:41.899098 8228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:41.900581 8214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:07:41.900722 8214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "b0d481779c1942348e69975f497d73f6"
format_stamp: "Formatted at 2026-09-27 07:07:41 on dist-test-slave-8vqf"
server_key: "8705f1fb21636e301963c9513e1bc206"
server_key_iv: "401b1e38ada272b8c67920d4996ae9e1"
server_key_version: "encryptionkey@0"
I20260927 07:07:41.900909 8214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:41.906744 8214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:41.907536 8214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:41.907761 8214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:41.908654 8214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:41.910036 8214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:41.910120 8214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:41.910205 8214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:41.910259 8214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:41.945174 8214 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:44185
I20260927 07:07:41.945277 8340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:44185 every 8 connection(s)
I20260927 07:07:41.946624 8214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:41.954798 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8214
I20260927 07:07:41.955060 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:41.955972 30785 external_mini_cluster.cc:1428] Setting key ad2fdbd10b49441a3349e37b1431e82c
I20260927 07:07:41.956275 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:41.949823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:43031 (local address 127.30.16.126:38575)
0927 07:07:41.950248 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:41.950252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:41.951661 (+ 1409us) server_negotiation.cc:407] Connection header received
0927 07:07:41.953547 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:41.953552 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:41.953608 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:41.953710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:41.955455 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:41.955459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:41.955462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:41.955479 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:41.955520 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:41.955532 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:41.955638 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:41.955920 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 07:07:41.955971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":195,"threads_started":1}
I20260927 07:07:41.957679 8344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:41.950466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38575 (local address 127.30.16.65:43031)
0927 07:07:41.951488 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:41.951533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:41.953237 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:41.954012 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:41.954022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:41.954128 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:41.955065 (+ 937us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:41.955069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:41.955129 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:41.955262 (+ 133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:41.955642 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:41.955655 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:41.955664 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 07:07:41.956001 (+ 337us) client_negotiation.cc:240] Negotiation successful
0927 07:07:41.956323 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":287,"threads_started":1}
I20260927 07:07:41.959308 8341 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38575
I20260927 07:07:41.959326 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--builtin_ntp_servers=127.30.16.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 07:07:41.959654 8341 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:41.960379 8341 heartbeater.cc:511] Master 127.30.16.126:38575 requested a full tablet report, sending...
I20260927 07:07:41.962873 8155 ts_manager.cc:194] Registered new tserver with Master: b0d481779c1942348e69975f497d73f6 (127.30.16.65:44185)
W20260927 07:07:42.295082 8345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:42.295368 8345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:42.295538 8345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:42.303272 8345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:42.303387 8345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:42.303556 8345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:42.312968 8345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:42.314379 8345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:42.316198 8345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:42.330283 8351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:42.330638 8353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:42.330682 8350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:42.331275 8345 server_base.cc:1034] running on GCE node
I20260927 07:07:42.331907 8345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:42.332994 8345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:42.334235 8345 hybrid_clock.cc:648] HybridClock initialized: now 1790492862334183 us; error 47 us; skew 500 ppm
I20260927 07:07:42.334538 8345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:42.337123 8345 webserver.cc:460] Webserver started at http://127.30.16.66:42065/ using document root <none> and password file <none>
I20260927 07:07:42.337810 8345 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:42.337896 8345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:42.338171 8345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:42.341076 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "0dffde9cda8443979591d7255eba6883"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "87595cf300af8c56f5593b4624558f6a"
server_key_iv: "b50aed577515055453c1edb6eee22d58"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.341890 8345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "0dffde9cda8443979591d7255eba6883"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "87595cf300af8c56f5593b4624558f6a"
server_key_iv: "b50aed577515055453c1edb6eee22d58"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.347404 8345 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 07:07:42.351038 8359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.352526 8345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:07:42.352697 8345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "0dffde9cda8443979591d7255eba6883"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "87595cf300af8c56f5593b4624558f6a"
server_key_iv: "b50aed577515055453c1edb6eee22d58"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.352932 8345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:42.358951 8345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:42.359773 8345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:42.360051 8345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:42.361135 8345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:42.362682 8345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:42.362790 8345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.362906 8345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:42.362998 8345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.397465 8345 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:41783
I20260927 07:07:42.397554 8471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:41783 every 8 connection(s)
I20260927 07:07:42.398875 8345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:42.404784 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8345
I20260927 07:07:42.404970 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:42.405514 30785 external_mini_cluster.cc:1428] Setting key ad7376d92a85a67cdf73116c0e7fa540
I20260927 07:07:42.408586 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.401933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49561 (local address 127.30.16.126:38575)
0927 07:07:42.402170 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:42.402176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:42.403860 (+ 1684us) server_negotiation.cc:407] Connection header received
0927 07:07:42.405972 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:42.405978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:42.406062 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:42.406182 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:42.407743 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:42.407749 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:42.407753 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:42.407773 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:42.407810 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:42.407826 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:42.407947 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:42.408198 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 07:07:42.408261 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 07:07:42.408782 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--builtin_ntp_servers=127.30.16.84:45725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 07:07:42.409811 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.402533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38575 (local address 127.30.16.66:49561)
0927 07:07:42.403690 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:42.403742 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:42.405584 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:42.406424 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:42.406433 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:42.406601 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:42.407389 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:42.407394 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:42.407440 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:42.407559 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:42.407947 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:42.407957 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:42.407965 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 07:07:42.408255 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 07:07:42.408528 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":330,"threads_started":1}
I20260927 07:07:42.411834 8472 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38575
I20260927 07:07:42.412158 8472 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:42.412846 8472 heartbeater.cc:511] Master 127.30.16.126:38575 requested a full tablet report, sending...
I20260927 07:07:42.414773 8155 ts_manager.cc:194] Registered new tserver with Master: 0dffde9cda8443979591d7255eba6883 (127.30.16.66:41783)
W20260927 07:07:42.750027 8475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:42.750325 8475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:42.750487 8475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:42.758090 8475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:42.758193 8475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:42.758378 8475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:42.767750 8475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:42.769138 8475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:42.770991 8475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:42.784827 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:42.784904 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:42.784868 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:42.791150 8475 server_base.cc:1034] running on GCE node
I20260927 07:07:42.791862 8475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:42.793001 8475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:42.794235 8475 hybrid_clock.cc:648] HybridClock initialized: now 1790492862794171 us; error 69 us; skew 500 ppm
I20260927 07:07:42.794529 8475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:07:42.797124 8475 webserver.cc:460] Webserver started at http://127.30.16.67:42785/ using document root <none> and password file <none>
I20260927 07:07:42.797822 8475 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:42.797910 8475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:42.798192 8475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:42.801172 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "1f39eb2a2d59451ea4be90880ae4327b"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "b46813687cc01dfd72516b89ae5b4d7b"
server_key_iv: "66827bb2b2bf0adf75b71c219a8bbce5"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.801923 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "1f39eb2a2d59451ea4be90880ae4327b"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "b46813687cc01dfd72516b89ae5b4d7b"
server_key_iv: "66827bb2b2bf0adf75b71c219a8bbce5"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.807427 8475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:07:42.810844 8489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.812294 8475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 07:07:42.812474 8475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "1f39eb2a2d59451ea4be90880ae4327b"
format_stamp: "Formatted at 2026-09-27 07:07:42 on dist-test-slave-8vqf"
server_key: "b46813687cc01dfd72516b89ae5b4d7b"
server_key_iv: "66827bb2b2bf0adf75b71c219a8bbce5"
server_key_version: "encryptionkey@0"
I20260927 07:07:42.812707 8475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:42.818720 8475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:42.819535 8475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:42.819808 8475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:42.820752 8475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:42.822186 8475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:42.822309 8475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.822446 8475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:42.822536 8475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:42.856657 8475 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:43105
I20260927 07:07:42.856808 8601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:43105 every 8 connection(s)
I20260927 07:07:42.858232 8475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:42.864432 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8475
I20260927 07:07:42.864631 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:42.865177 30785 external_mini_cluster.cc:1428] Setting key 9e42394256ea37d7587b41a384716751
I20260927 07:07:42.867823 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.861012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:55207 (local address 127.30.16.126:38575)
0927 07:07:42.861234 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:42.861243 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:42.862966 (+ 1723us) server_negotiation.cc:407] Connection header received
0927 07:07:42.865100 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:42.865105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:42.865162 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:42.865241 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:42.866879 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:42.866884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:42.866888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:42.866905 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:42.866938 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:42.866952 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:42.867120 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:42.867399 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 07:07:42.867467 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 07:07:42.868875 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.861604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38575 (local address 127.30.16.67:55207)
0927 07:07:42.862741 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:42.862784 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:42.864748 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:42.865566 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:42.865579 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:42.865714 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:42.866442 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:07:42.866444 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:42.866483 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:07:42.866661 (+ 178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:42.867106 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:42.867114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:42.867121 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 07:07:42.867427 (+ 306us) client_negotiation.cc:240] Negotiation successful
0927 07:07:42.867657 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":360,"threads_started":1}
I20260927 07:07:42.870258 8602 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38575
I20260927 07:07:42.870664 8602 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:42.871313 8602 heartbeater.cc:511] Master 127.30.16.126:38575 requested a full tablet report, sending...
I20260927 07:07:42.872766 8155 ts_manager.cc:194] Registered new tserver with Master: 1f39eb2a2d59451ea4be90880ae4327b (127.30.16.67:43105)
I20260927 07:07:42.880950 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 07:07:42.887286 8343 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:07:42.885519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58512 (local address 127.30.16.126:38575)
0927 07:07:42.885727 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:42.885735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:42.886215 (+ 480us) server_negotiation.cc:407] Connection header received
0927 07:07:42.886345 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:42.886348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:42.886402 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:42.886503 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:42.886980 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58512: BlockingRecv error: recv got EOF from 127.0.0.1:58512 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:07:42.887984 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8214
I20260927 07:07:42.891628 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.889013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.30.16.65:44185)
0927 07:07:42.889577 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:42.889584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:42.889611 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:07:42.889668 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:42.889674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:42.889891 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:42.890062 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:42.890944 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:42.890951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:42.890963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:42.890979 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:42.891019 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:42.891033 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:42.891137 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:42.891230 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 07:07:42.891335 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
I20260927 07:07:42.893052 8336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:42.973474 8341 heartbeater.cc:503] Master 127.30.16.126:38575 was elected leader, sending a full tablet report...
I20260927 07:07:42.975521 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8214
I20260927 07:07:42.992811 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8345
I20260927 07:07:42.996831 8613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:42.993853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.30.16.66:41783)
0927 07:07:42.994428 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:42.994435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:42.994464 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:07:42.994530 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:42.994536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:42.994852 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:42.995001 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:42.995993 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:42.996000 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:42.996013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:42.996031 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:42.996074 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:42.996089 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:42.996200 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:42.996435 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 07:07:42.996557 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260927 07:07:42.998278 8467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:43.079988 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8345
I20260927 07:07:43.095829 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8475
I20260927 07:07:43.099995 8615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:43.096824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.30.16.67:43105)
0927 07:07:43.097492 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:43.097498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:43.097532 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:07:43.097594 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:43.097600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:43.097829 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:43.098002 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:43.099261 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:43.099268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:43.099279 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:07:43.099296 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:43.099333 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:07:43.099348 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:43.099529 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:43.099603 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 07:07:43.099700 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":351,"threads_started":1}
I20260927 07:07:43.101282 8597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:43.186053 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8475
I20260927 07:07:43.202791 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8124
I20260927 07:07:43.204010 8185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:43.307371 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8124
2026-09-27T07:07:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2407 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:43 dist-test-slave-8vqf krb5kdc[8620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:43 dist-test-slave-8vqf krb5kdc[8620](info): set up 2 sockets
Sep 27 07:07:43 dist-test-slave-8vqf krb5kdc[8620](info): commencing operation
krb5kdc: starting...
W20260927 07:07:44.411661 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:07:44 dist-test-slave-8vqf krb5kdc[8620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:07:44.651336 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38483
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:45.002714 8633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:45.003167 8633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:45.003325 8633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:45.011281 8633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:07:45.011466 8633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:45.011554 8633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:45.011659 8633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:07:45.011729 8633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:07:45.022440 8633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:38483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:45.023998 8633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:45.025348 8633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:45.037551 8638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:45.042865 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:45.042960 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:45.042960 8633 server_base.cc:1034] running on GCE node
I20260927 07:07:45.044051 8633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:45.045503 8633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:45.046665 8633 hybrid_clock.cc:648] HybridClock initialized: now 1790492865046611 us; error 36 us; skew 500 ppm
I20260927 07:07:45.047030 8633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:45 dist-test-slave-8vqf krb5kdc[8620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492865, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:45.050288 8633 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:07:45.052017 8633 webserver.cc:460] Webserver started at http://127.30.16.126:37125/ using document root <none> and password file <none>
I20260927 07:07:45.052737 8633 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:45.052834 8633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:45.053140 8633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:45.056187 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "e69d0a3a21684dcaa643f5887675ae4c"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "2dcbfb6ded73884ed11df95dc9c4dfdc"
server_key_iv: "8230807867cd19f2f922ab3fa7b297cc"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.056949 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "e69d0a3a21684dcaa643f5887675ae4c"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "2dcbfb6ded73884ed11df95dc9c4dfdc"
server_key_iv: "8230807867cd19f2f922ab3fa7b297cc"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.062251 8633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:07:45.066104 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:45.067581 8633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:07:45.067756 8633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "e69d0a3a21684dcaa643f5887675ae4c"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "2dcbfb6ded73884ed11df95dc9c4dfdc"
server_key_iv: "8230807867cd19f2f922ab3fa7b297cc"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.067950 8633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:45.110739 8633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:45.115056 8633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:45.115350 8633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:45.134073 8633 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:38483
I20260927 07:07:45.134109 8699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:38483 every 8 connection(s)
I20260927 07:07:45.135488 8633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:07:45.139227 8700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:45.144767 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8633
I20260927 07:07:45.145040 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:07:45.145686 30785 external_mini_cluster.cc:1428] Setting key 07e1d147c759a264fb37d377e3eef5f6
I20260927 07:07:45.150712 8700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: Bootstrap starting.
I20260927 07:07:45.154500 8700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:45.155834 8700 log.cc:824] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:45.159489 8700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: No bootstrap required, opened a new log
Sep 27 07:07:45 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:45.167704 8700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } }
I20260927 07:07:45.168081 8700 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:45.168229 8700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69d0a3a21684dcaa643f5887675ae4c, State: Initialized, Role: FOLLOWER
I20260927 07:07:45.168980 8700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } }
I20260927 07:07:45.169268 8700 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:07:45.169406 8700 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:07:45.169584 8700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:45.170694 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:45.147809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.30.16.126:38483)
0927 07:07:45.148724 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:45.148737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:45.148829 (+ 92us) server_negotiation.cc:407] Connection header received
0927 07:07:45.150517 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:45.150541 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:45.151047 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:45.151414 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:45.153274 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.154820 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.156683 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.157687 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.157826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:45.162696 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:45.162708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:45.162723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:45.162762 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:45.165908 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:45.166686 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:45.166693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:45.166700 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:45.166868 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:45.167259 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:45.167268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:45.167270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:45.167929 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:45.168257 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:45.168747 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 07:07:45.169214 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":399,"threads_started":1}
I20260927 07:07:45.172369 8700 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } }
I20260927 07:07:45.172916 8700 leader_election.cc:302] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69d0a3a21684dcaa643f5887675ae4c; no voters:
I20260927 07:07:45.173574 8700 leader_election.cc:288] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:07:45.173727 8705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:45.174713 8705 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [term 1 LEADER]: Becoming Leader. State: Replica: e69d0a3a21684dcaa643f5887675ae4c, State: Running, Role: LEADER
I20260927 07:07:45.175252 8705 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } }
I20260927 07:07:45.175698 8700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:07:45.178076 8706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69d0a3a21684dcaa643f5887675ae4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } } }
I20260927 07:07:45.178416 8706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:45.178179 8707 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69d0a3a21684dcaa643f5887675ae4c. Latest consensus state: current_term: 1 leader_uuid: "e69d0a3a21684dcaa643f5887675ae4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69d0a3a21684dcaa643f5887675ae4c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38483 } } }
I20260927 07:07:45.178612 8707 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c [sys.catalog]: This master's current role is: LEADER
I20260927 07:07:45.181550 8711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:07:45.187201 8711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:07:45.193848 8722 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:07:45.197742 8711 catalog_manager.cc:1347] Generated new cluster ID: 72775454d77745599daa2766a0d3b697
I20260927 07:07:45.197867 8711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:07:45.222997 8711 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:07:45.224385 8711 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:07:45.237315 8711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: Generated new TSK 0
I20260927 07:07:45.238404 8711 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:07:45.307000 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:45.673276 8726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:45.673568 8726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:45.673748 8726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:45.681303 8726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:45.681421 8726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:45.681630 8726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:07:45.691929 8726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:45.693264 8726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:45.694746 8726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:45.708091 8732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:45.708287 8734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:45.708705 8731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:45.708868 8726 server_base.cc:1034] running on GCE node
I20260927 07:07:45.709540 8726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:45.710536 8726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:45.711812 8726 hybrid_clock.cc:648] HybridClock initialized: now 1790492865711753 us; error 47 us; skew 500 ppm
I20260927 07:07:45.712146 8726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:45 dist-test-slave-8vqf krb5kdc[8620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492865, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:45.715665 8726 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:07:45.717183 8726 webserver.cc:460] Webserver started at http://127.30.16.65:45807/ using document root <none> and password file <none>
I20260927 07:07:45.717914 8726 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:45.718065 8726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:45.718376 8726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:45.721422 8726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "28a491bea55744f489d28c775c031766"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "170e8c60e80b7106e71495fd6ecc2a8c"
server_key_iv: "dcc90040d40ea96dc148557b32e2dccd"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.722204 8726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "28a491bea55744f489d28c775c031766"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "170e8c60e80b7106e71495fd6ecc2a8c"
server_key_iv: "dcc90040d40ea96dc148557b32e2dccd"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.727689 8726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 07:07:45.731308 8741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:45.732817 8726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:07:45.733006 8726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "28a491bea55744f489d28c775c031766"
format_stamp: "Formatted at 2026-09-27 07:07:45 on dist-test-slave-8vqf"
server_key: "170e8c60e80b7106e71495fd6ecc2a8c"
server_key_iv: "dcc90040d40ea96dc148557b32e2dccd"
server_key_version: "encryptionkey@0"
I20260927 07:07:45.733220 8726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:45.774201 8726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:45.778791 8726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:45.779111 8726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:45.780140 8726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:45.781492 8726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:45.781610 8726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:45.781754 8726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:45.781828 8726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:45.817391 8726 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:38541
I20260927 07:07:45.817454 8853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:38541 every 8 connection(s)
I20260927 07:07:45.818732 8726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:07:45.827428 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8726
I20260927 07:07:45.827682 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:07:45.828435 30785 external_mini_cluster.cc:1428] Setting key 3d24a64ac2215b2ccd3ebfd744e600a6
Sep 27 07:07:45 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492865, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:45.840051 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:45.821533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:48761 (local address 127.30.16.126:38483)
0927 07:07:45.822040 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:45.822045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:45.823280 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 07:07:45.825101 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:45.825105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:45.825181 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:45.825282 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:45.826959 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.828097 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.829802 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.830255 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.830397 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:45.834172 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:45.834178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:45.834182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:45.834213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:45.836849 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:45.837610 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:45.837614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:45.837617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:45.837796 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:45.838243 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:45.838248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:45.838251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:45.838517 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:45.838710 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:45.839421 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 07:07:45.839649 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":233,"threads_started":1}
I20260927 07:07:45.841092 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:45.822201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38483 (local address 127.30.16.65:48761)
0927 07:07:45.823117 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:45.823157 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:45.824827 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:45.825540 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:45.825549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:45.826080 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:45.826745 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:45.826770 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.828297 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.828302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:45.829632 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:45.829643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:45.830495 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:45.830502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:45.830774 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:45.831541 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:45.831564 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:45.833925 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:45.836985 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:45.836996 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:45.837014 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:45.837464 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:45.837920 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:45.837926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:45.837929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:45.838123 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:45.838677 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:45.838685 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:45.839037 (+ 352us) client_negotiation.cc:769] Sending connection context
0927 07:07:45.839316 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 07:07:45.839911 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":465,"threads_started":1}
I20260927 07:07:45.842782 8854 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38483
I20260927 07:07:45.843144 8854 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:45.843892 8854 heartbeater.cc:511] Master 127.30.16.126:38483 requested a full tablet report, sending...
I20260927 07:07:45.846313 8665 ts_manager.cc:194] Registered new tserver with Master: 28a491bea55744f489d28c775c031766 (127.30.16.65:38541)
I20260927 07:07:45.848085 8665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:48761
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:07:45.885905 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:46.221680 8860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:46.221990 8860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:46.222096 8860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:46.229462 8860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:46.229578 8860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:46.229691 8860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:07:46.239886 8860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:46.241173 8860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:46.242507 8860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:46.256011 8865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:46.256160 8866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:46.256126 8868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:46.257212 8860 server_base.cc:1034] running on GCE node
I20260927 07:07:46.257874 8860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:46.258925 8860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:46.260130 8860 hybrid_clock.cc:648] HybridClock initialized: now 1790492866260082 us; error 38 us; skew 500 ppm
I20260927 07:07:46.260463 8860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:46 dist-test-slave-8vqf krb5kdc[8620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:46.263702 8860 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:07:46.265241 8860 webserver.cc:460] Webserver started at http://127.30.16.66:33347/ using document root <none> and password file <none>
I20260927 07:07:46.265934 8860 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:46.266039 8860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:46.266286 8860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:46.269269 8860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "9709e705b38b4eea9ad4757301e02cb2"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "325abdf1b22da1e3fdfd2af7f5735c11"
server_key_iv: "be7471c48dc7a82037f62b897232613a"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.269972 8860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "9709e705b38b4eea9ad4757301e02cb2"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "325abdf1b22da1e3fdfd2af7f5735c11"
server_key_iv: "be7471c48dc7a82037f62b897232613a"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.275560 8860 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 07:07:46.279235 8875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.280711 8860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:07:46.280902 8860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "9709e705b38b4eea9ad4757301e02cb2"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "325abdf1b22da1e3fdfd2af7f5735c11"
server_key_iv: "be7471c48dc7a82037f62b897232613a"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.281124 8860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:46.299146 8860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:46.303457 8860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:46.303787 8860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:46.304778 8860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:46.306207 8860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:46.306313 8860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.306471 8860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:46.306553 8860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.341687 8860 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:35723
I20260927 07:07:46.341773 8987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:35723 every 8 connection(s)
I20260927 07:07:46.343197 8860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:07:46.349202 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8860
I20260927 07:07:46.349403 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:07:46.349862 30785 external_mini_cluster.cc:1428] Setting key 187097db98078bc9d7d700dddf59763b
Sep 27 07:07:46 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:46.365964 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.345815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49351 (local address 127.30.16.126:38483)
0927 07:07:46.346263 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.346268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.347417 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 07:07:46.349494 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.349500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.349610 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.349736 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:46.351632 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.352678 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.354435 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.355184 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.355321 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.359558 (+ 4237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:46.359563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:46.359566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:46.359603 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:46.362309 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.363447 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.363451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.363454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.363587 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.364134 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.364137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.364139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.364401 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:46.364524 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.365521 (+ 997us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.365692 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":234,"threads_started":1}
I20260927 07:07:46.367851 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.346433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38483 (local address 127.30.16.66:49351)
0927 07:07:46.347250 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:46.347291 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:46.349155 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:46.350044 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:46.350054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:46.350809 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:46.351421 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.351439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.352852 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.352856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.354275 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.354288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.355425 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.355431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.355665 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.356434 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:46.356463 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:46.359284 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:46.362481 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:46.362495 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:46.362520 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:46.363312 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:46.363707 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:46.363716 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:46.363723 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:46.364019 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:46.364540 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:46.364552 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:46.365104 (+ 552us) client_negotiation.cc:769] Sending connection context
0927 07:07:46.365444 (+ 340us) client_negotiation.cc:240] Negotiation successful
0927 07:07:46.366113 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":278,"threads_started":1}
I20260927 07:07:46.369553 8988 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38483
I20260927 07:07:46.369894 8988 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:46.370841 8988 heartbeater.cc:511] Master 127.30.16.126:38483 requested a full tablet report, sending...
I20260927 07:07:46.372581 8665 ts_manager.cc:194] Registered new tserver with Master: 9709e705b38b4eea9ad4757301e02cb2 (127.30.16.66:35723)
I20260927 07:07:46.373438 8665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:49351
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:07:46.406793 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:07:46.745105 8994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:07:46.745400 8994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:07:46.745549 8994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:07:46.753341 8994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:07:46.753496 8994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:07:46.753677 8994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:07:46.763752 8994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:38483
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:07:46.765129 8994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:07:46.766484 8994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:07:46.780679 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:46.780753 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:07:46.780717 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:07:46.781536 8994 server_base.cc:1034] running on GCE node
I20260927 07:07:46.782171 8994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:07:46.783231 8994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:07:46.784488 8994 hybrid_clock.cc:648] HybridClock initialized: now 1790492866784434 us; error 42 us; skew 500 ppm
I20260927 07:07:46.784813 8994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:07:46 dist-test-slave-8vqf krb5kdc[8620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:07:46.788187 8994 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:07:46.789664 8994 webserver.cc:460] Webserver started at http://127.30.16.67:38275/ using document root <none> and password file <none>
I20260927 07:07:46.790393 8994 fs_manager.cc:359] Metadata directory not provided
I20260927 07:07:46.790587 8994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:07:46.790900 8994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:07:46.793856 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "85e538399b96425a8f5086b41b7a2e7a"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "139916ea1b713caf9deb5122aeca8b33"
server_key_iv: "d9694be1a023b391fb2788ab7277bd0d"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.794760 8994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "85e538399b96425a8f5086b41b7a2e7a"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "139916ea1b713caf9deb5122aeca8b33"
server_key_iv: "d9694be1a023b391fb2788ab7277bd0d"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.800123 8994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 07:07:46.803707 9009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.805048 8994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:07:46.805240 8994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "85e538399b96425a8f5086b41b7a2e7a"
format_stamp: "Formatted at 2026-09-27 07:07:46 on dist-test-slave-8vqf"
server_key: "139916ea1b713caf9deb5122aeca8b33"
server_key_iv: "d9694be1a023b391fb2788ab7277bd0d"
server_key_version: "encryptionkey@0"
I20260927 07:07:46.805465 8994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:07:46.826925 8994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:07:46.831458 8994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:07:46.831785 8994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:07:46.832793 8994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:07:46.834146 8994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:07:46.834255 8994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.834405 8994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:07:46.834478 8994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:07:46.851537 8854 heartbeater.cc:503] Master 127.30.16.126:38483 was elected leader, sending a full tablet report...
I20260927 07:07:46.868136 8994 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:40167
I20260927 07:07:46.868216 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:40167 every 8 connection(s)
I20260927 07:07:46.869552 8994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:07:46.869895 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 8994
I20260927 07:07:46.870114 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:07:46.870749 30785 external_mini_cluster.cc:1428] Setting key 39b33cc0315b1685b7c17b0884e0a119
Sep 27 07:07:46 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492866, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:07:46.890885 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.872824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:40379 (local address 127.30.16.126:38483)
0927 07:07:46.873270 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.873276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.874418 (+ 1142us) server_negotiation.cc:407] Connection header received
0927 07:07:46.876453 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.876458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.876564 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.876678 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:46.878215 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.879049 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.880659 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.881053 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.881186 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.884902 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:46.884909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:46.884914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:46.884953 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:46.888102 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.888819 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.888823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.888825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.888935 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.889293 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.889296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.889298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.889578 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:46.889694 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.890384 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.890607 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":222,"threads_started":1}
I20260927 07:07:46.891902 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.873388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:38483 (local address 127.30.16.67:40379)
0927 07:07:46.874264 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:46.874319 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:46.876146 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:46.876939 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:46.876947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:46.877462 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:07:46.878046 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.878064 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.879214 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.879220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.880482 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.880495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.881299 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.881308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.881544 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.882194 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:07:46.882220 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:07:46.884607 (+ 2387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:07:46.888268 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:46.888275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:46.888291 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:46.888722 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:46.889033 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:07:46.889036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:07:46.889038 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:07:46.889204 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:07:46.889686 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:07:46.889692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:07:46.890068 (+ 376us) client_negotiation.cc:769] Sending connection context
0927 07:07:46.890310 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 07:07:46.890779 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":324,"threads_started":1}
I20260927 07:07:46.893263 9122 heartbeater.cc:347] Connected to a master server at 127.30.16.126:38483
I20260927 07:07:46.893575 9122 heartbeater.cc:464] Registering TS with master...
I20260927 07:07:46.894336 9122 heartbeater.cc:511] Master 127.30.16.126:38483 requested a full tablet report, sending...
I20260927 07:07:46.896003 8665 ts_manager.cc:194] Registered new tserver with Master: 85e538399b96425a8f5086b41b7a2e7a (127.30.16.67:40167)
I20260927 07:07:46.896854 8665 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:40379
I20260927 07:07:46.899622 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:07:46.918409 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.30.16.126:38483)
0927 07:07:46.904202 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.904206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.904646 (+ 440us) server_negotiation.cc:407] Connection header received
0927 07:07:46.904867 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.904872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.904936 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.905040 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:46.906482 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.907288 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.908565 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.909034 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.909183 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.912785 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:46.912791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:46.912795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:46.912833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:46.915649 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.916300 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.916304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.916307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.916455 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:46.916911 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:46.916915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:46.916917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:46.917334 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:46.917542 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.917926 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.918122 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:07:46.923004 8665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:07:46.930416 8665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:07:46.966091 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.943769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:35723 (local address 127.0.0.1:42812)
0927 07:07:46.952933 (+ 9164us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:46.952995 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:46.954393 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:46.956245 (+ 1852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:46.956253 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:46.956298 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:46.956817 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.956833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.959919 (+ 3086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.959931 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.963092 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.963109 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.965052 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.965068 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.965289 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.965364 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 07:07:46.965511 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 07:07:46.965706 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8763,"thread_start_us":12200,"threads_started":1}
I20260927 07:07:46.966115 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.944310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.30.16.66:35723)
0927 07:07:46.944903 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.944918 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.953198 (+ 8280us) server_negotiation.cc:407] Connection header received
0927 07:07:46.955197 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.955212 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.955624 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.955847 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:46.957900 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.959698 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.963285 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.964826 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.964971 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.965174 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.965570 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.965746 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":271,"threads_started":1}
I20260927 07:07:46.968780 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.943810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38541 (local address 127.0.0.1:43980)
0927 07:07:46.956284 (+ 12474us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:46.956303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:46.956417 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:46.959390 (+ 2973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:46.959399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:46.959434 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:46.960011 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.960026 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.963691 (+ 3665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.963697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.965780 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.965792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.968254 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.968258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.968378 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.968400 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:07:46.968458 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 07:07:46.968551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12347,"thread_start_us":19609,"threads_started":1}
I20260927 07:07:46.969007 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.943817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.30.16.65:38541)
0927 07:07:46.958271 (+ 14454us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.958284 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.958324 (+ 40us) server_negotiation.cc:407] Connection header received
0927 07:07:46.958443 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.958456 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.958841 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.959082 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:46.960284 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.963511 (+ 3227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.965967 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.968045 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.968201 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.968425 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.968523 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.968687 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14229,"thread_start_us":14020,"threads_started":1}
I20260927 07:07:46.970839 8923 tablet_service.cc:1467] Processing CreateTablet for tablet d66297e796a44c6e84b25b8975e82a80 (DEFAULT_TABLE table=test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:46.972415 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66297e796a44c6e84b25b8975e82a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:46.972733 8789 tablet_service.cc:1467] Processing CreateTablet for tablet d66297e796a44c6e84b25b8975e82a80 (DEFAULT_TABLE table=test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:46.973341 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.947393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:40167 (local address 127.0.0.1:43876)
0927 07:07:46.963696 (+ 16303us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:46.963711 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:46.963861 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:46.964421 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:46.964425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:46.964449 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:46.964875 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.964885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.966860 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.966865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.970366 (+ 3501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:46.970380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.972575 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.972581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:46.972758 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.972789 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:07:46.972878 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:07:46.973010 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16173,"thread_start_us":4950,"threads_started":1}
I20260927 07:07:46.974105 8789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66297e796a44c6e84b25b8975e82a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:46.974157 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:46.947685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.30.16.67:40167)
0927 07:07:46.963131 (+ 15446us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:46.963142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:46.963816 (+ 674us) server_negotiation.cc:407] Connection header received
0927 07:07:46.964035 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:46.964041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:46.964238 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:46.964441 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:46.965107 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.966697 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.970533 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:46.972070 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:46.972198 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:46.972419 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:46.973613 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0927 07:07:46.973779 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15247,"thread_start_us":15085,"threads_started":1}
I20260927 07:07:46.982874 9057 tablet_service.cc:1467] Processing CreateTablet for tablet d66297e796a44c6e84b25b8975e82a80 (DEFAULT_TABLE table=test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:07:46.984468 9057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d66297e796a44c6e84b25b8975e82a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:07:46.984521 9141 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Bootstrap starting.
I20260927 07:07:46.988384 9142 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Bootstrap starting.
I20260927 07:07:46.989349 9141 tablet_bootstrap.cc:654] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:46.990759 9141 log.cc:824] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:46.992015 9142 tablet_bootstrap.cc:654] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:46.993043 9142 log.cc:824] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:46.994097 9141 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: No bootstrap required, opened a new log
I20260927 07:07:46.994395 9141 ts_tablet_manager.cc:1397] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260927 07:07:46.995638 9142 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: No bootstrap required, opened a new log
I20260927 07:07:46.995899 9142 ts_tablet_manager.cc:1397] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 07:07:46.999840 9145 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Bootstrap starting.
I20260927 07:07:47.003732 9145 tablet_bootstrap.cc:654] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Neither blocks nor log segments found. Creating new log.
I20260927 07:07:47.004654 9141 raft_consensus.cc:348] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.005048 9141 raft_consensus.cc:374] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:47.005067 9145 log.cc:824] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Log is configured to *not* fsync() on all Append() calls
I20260927 07:07:47.005164 9141 raft_consensus.cc:729] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9709e705b38b4eea9ad4757301e02cb2, State: Initialized, Role: FOLLOWER
I20260927 07:07:47.004905 9142 raft_consensus.cc:348] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.005322 9142 raft_consensus.cc:374] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:47.005425 9142 raft_consensus.cc:729] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a491bea55744f489d28c775c031766, State: Initialized, Role: FOLLOWER
I20260927 07:07:47.005893 9141 consensus_queue.cc:260] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.006130 9142 consensus_queue.cc:260] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.007160 8988 heartbeater.cc:503] Master 127.30.16.126:38483 was elected leader, sending a full tablet report...
I20260927 07:07:47.007898 9141 ts_tablet_manager.cc:1428] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Time spent starting tablet: real 0.013s user 0.007s sys 0.007s
I20260927 07:07:47.009660 9142 ts_tablet_manager.cc:1428] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 07:07:47.014525 9145 tablet_bootstrap.cc:492] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: No bootstrap required, opened a new log
I20260927 07:07:47.014900 9145 ts_tablet_manager.cc:1397] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260927 07:07:47.022012 9145 raft_consensus.cc:348] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.022404 9145 raft_consensus.cc:374] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:07:47.022527 9145 raft_consensus.cc:729] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85e538399b96425a8f5086b41b7a2e7a, State: Initialized, Role: FOLLOWER
I20260927 07:07:47.023211 9145 consensus_queue.cc:260] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.024269 9122 heartbeater.cc:503] Master 127.30.16.126:38483 was elected leader, sending a full tablet report...
I20260927 07:07:47.025019 9145 ts_tablet_manager.cc:1428] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
W20260927 07:07:47.071286 8855 tablet.cc:2369] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:47.095151 8989 tablet.cc:2369] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:07:47.121317 9123 tablet.cc:2369] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:07:47.225287 9147 raft_consensus.cc:482] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:07:47.225579 9147 raft_consensus.cc:504] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.227368 9147 leader_election.cc:288] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a491bea55744f489d28c775c031766 (127.30.16.65:38541), 85e538399b96425a8f5086b41b7a2e7a (127.30.16.67:40167)
I20260927 07:07:47.234350 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.227907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:57841 (local address 127.30.16.65:38541)
0927 07:07:47.228118 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.228126 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.228910 (+ 784us) server_negotiation.cc:407] Connection header received
0927 07:07:47.229140 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.229150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.229210 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.229307 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:47.229882 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.230684 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.232572 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.233372 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.233566 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.233649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.233968 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.234095 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:07:47.234369 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.227822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:59905 (local address 127.30.16.67:40167)
0927 07:07:47.228042 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.228053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.229020 (+ 967us) server_negotiation.cc:407] Connection header received
0927 07:07:47.229204 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.229210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.229288 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.229381 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:07:47.230037 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.230824 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.232583 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.233377 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.233566 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.233653 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.233977 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.234117 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:07:47.234596 9150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.227797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:38541 (local address 127.30.16.66:57841)
0927 07:07:47.228789 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:47.228805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:47.228989 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:47.229333 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:47.229339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:47.229363 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:47.229728 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:47.229737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.230811 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.230818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:47.232406 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:47.232418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.233617 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.233625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:47.233781 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.233830 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 07:07:47.233940 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 07:07:47.234071 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":66,"thread_start_us":488,"threads_started":1}
I20260927 07:07:47.234619 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.227831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:40167 (local address 127.30.16.66:59905)
0927 07:07:47.228894 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 07:07:47.228908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:07:47.229060 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:07:47.229412 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:07:47.229416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:07:47.229445 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:07:47.229876 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:47.229884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.230959 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.230974 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:47.232421 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:07:47.232433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.233654 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.233661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:07:47.233829 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.233851 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:07:47.233952 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 07:07:47.234074 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":4,"thread_start_us":487,"threads_started":1}
I20260927 07:07:47.235853 9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66297e796a44c6e84b25b8975e82a80" candidate_uuid: "9709e705b38b4eea9ad4757301e02cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e538399b96425a8f5086b41b7a2e7a" is_pre_election: true
I20260927 07:07:47.235853 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66297e796a44c6e84b25b8975e82a80" candidate_uuid: "9709e705b38b4eea9ad4757301e02cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a491bea55744f489d28c775c031766" is_pre_election: true
I20260927 07:07:47.236454 9077 raft_consensus.cc:2454] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9709e705b38b4eea9ad4757301e02cb2 in term 0.
I20260927 07:07:47.236452 8809 raft_consensus.cc:2454] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9709e705b38b4eea9ad4757301e02cb2 in term 0.
I20260927 07:07:47.237231 8877 leader_election.cc:302] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28a491bea55744f489d28c775c031766, 9709e705b38b4eea9ad4757301e02cb2; no voters:
I20260927 07:07:47.237630 9147 raft_consensus.cc:2781] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:07:47.237820 9147 raft_consensus.cc:482] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:07:47.237943 9147 raft_consensus.cc:3037] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:47.240222 9147 raft_consensus.cc:504] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.240831 9147 leader_election.cc:288] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [CANDIDATE]: Term 1 election: Requested vote from peers 28a491bea55744f489d28c775c031766 (127.30.16.65:38541), 85e538399b96425a8f5086b41b7a2e7a (127.30.16.67:40167)
I20260927 07:07:47.241379 9077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66297e796a44c6e84b25b8975e82a80" candidate_uuid: "9709e705b38b4eea9ad4757301e02cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85e538399b96425a8f5086b41b7a2e7a"
I20260927 07:07:47.241379 8809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d66297e796a44c6e84b25b8975e82a80" candidate_uuid: "9709e705b38b4eea9ad4757301e02cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a491bea55744f489d28c775c031766"
I20260927 07:07:47.241604 9077 raft_consensus.cc:3037] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:47.241604 8809 raft_consensus.cc:3037] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:07:47.244895 8809 raft_consensus.cc:2454] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9709e705b38b4eea9ad4757301e02cb2 in term 1.
I20260927 07:07:47.244906 9077 raft_consensus.cc:2454] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9709e705b38b4eea9ad4757301e02cb2 in term 1.
I20260927 07:07:47.245321 8878 leader_election.cc:302] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85e538399b96425a8f5086b41b7a2e7a, 9709e705b38b4eea9ad4757301e02cb2; no voters:
I20260927 07:07:47.245685 9147 raft_consensus.cc:2781] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:07:47.246097 9147 raft_consensus.cc:686] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 9709e705b38b4eea9ad4757301e02cb2, State: Running, Role: LEADER
I20260927 07:07:47.246627 9147 consensus_queue.cc:237] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } }
I20260927 07:07:47.251384 8665 catalog_manager.cc:5495] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9709e705b38b4eea9ad4757301e02cb2 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "9709e705b38b4eea9ad4757301e02cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9709e705b38b4eea9ad4757301e02cb2" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 } health_report { overall_health: UNKNOWN } } }
I20260927 07:07:47.327837 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.322538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.30.16.66:35723)
0927 07:07:47.322794 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.322800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.322832 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:07:47.322939 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.322943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.323033 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.323194 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:07:47.323568 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.324281 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.325627 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.326020 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.326154 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.326704 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:07:47.327236 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:07:47.327414 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.327465 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.327559 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:07:47.336437 9077 raft_consensus.cc:1261] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 1 FOLLOWER]: Refusing update from remote peer 9709e705b38b4eea9ad4757301e02cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:47.336428 8809 raft_consensus.cc:1261] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 1 FOLLOWER]: Refusing update from remote peer 9709e705b38b4eea9ad4757301e02cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:07:47.337441 9152 consensus_queue.cc:1035] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a491bea55744f489d28c775c031766" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 38541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:47.337791 9147 consensus_queue.cc:1035] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85e538399b96425a8f5086b41b7a2e7a" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 40167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:07:47.354632 9155 mvcc.cc:205] Tried to move back new op lower bound from 7333858784598941696 to 7333858784245944320. Current Snapshot: MvccSnapshot[applied={T|T < 7333858784598941696}]
I20260927 07:07:47.375069 8665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:47.375394 8665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:07:47.379191 8665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e69d0a3a21684dcaa643f5887675ae4c: Sending DeleteTablet for 3 replicas of tablet d66297e796a44c6e84b25b8975e82a80
I20260927 07:07:47.380380 8789 tablet_service.cc:1514] Processing DeleteTablet for tablet d66297e796a44c6e84b25b8975e82a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:47 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:43980
I20260927 07:07:47.380478 9057 tablet_service.cc:1514] Processing DeleteTablet for tablet d66297e796a44c6e84b25b8975e82a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:47 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:43876
I20260927 07:07:47.380505 8923 tablet_service.cc:1514] Processing DeleteTablet for tablet d66297e796a44c6e84b25b8975e82a80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:07:47 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:42812
I20260927 07:07:47.383335 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8726
I20260927 07:07:47.393066 9165 tablet_replica.cc:330] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: stopping tablet replica
I20260927 07:07:47.393487 9165 raft_consensus.cc:2229] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:47.393978 9165 raft_consensus.cc:2258] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:47.397430 9165 ts_tablet_manager.cc:1905] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:47.399284 9164 tablet_replica.cc:330] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: stopping tablet replica
I20260927 07:07:47.399773 9164 raft_consensus.cc:2229] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:07:47.400465 9164 raft_consensus.cc:2258] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:47.402630 9164 ts_tablet_manager.cc:1905] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:07:47.404663 9165 ts_tablet_manager.cc:1918] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:47.404820 9165 log.cc:1196] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/d66297e796a44c6e84b25b8975e82a80
I20260927 07:07:47.405400 9165 ts_tablet_manager.cc:1939] T d66297e796a44c6e84b25b8975e82a80 P 28a491bea55744f489d28c775c031766: Deleting consensus metadata
I20260927 07:07:47.407336 8650 catalog_manager.cc:4841] TS 28a491bea55744f489d28c775c031766 (127.30.16.65:38541): tablet d66297e796a44c6e84b25b8975e82a80 (table test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]) successfully deleted
I20260927 07:07:47.409221 9164 ts_tablet_manager.cc:1918] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:47.409422 9164 log.cc:1196] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/d66297e796a44c6e84b25b8975e82a80
I20260927 07:07:47.409957 9164 ts_tablet_manager.cc:1939] T d66297e796a44c6e84b25b8975e82a80 P 85e538399b96425a8f5086b41b7a2e7a: Deleting consensus metadata
I20260927 07:07:47.410966 9166 tablet_replica.cc:330] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: stopping tablet replica
I20260927 07:07:47.411437 9166 raft_consensus.cc:2229] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:07:47.411712 8651 catalog_manager.cc:4841] TS 85e538399b96425a8f5086b41b7a2e7a (127.30.16.67:40167): tablet d66297e796a44c6e84b25b8975e82a80 (table test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]) successfully deleted
I20260927 07:07:47.412129 9166 raft_consensus.cc:2258] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:07:47.413765 9166 ts_tablet_manager.cc:1905] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 07:07:47 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:07:47.419711 9166 ts_tablet_manager.cc:1918] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:07:47.419884 9166 log.cc:1196] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/d66297e796a44c6e84b25b8975e82a80
I20260927 07:07:47.420352 9166 ts_tablet_manager.cc:1939] T d66297e796a44c6e84b25b8975e82a80 P 9709e705b38b4eea9ad4757301e02cb2: Deleting consensus metadata
I20260927 07:07:47.422362 8652 catalog_manager.cc:4841] TS 9709e705b38b4eea9ad4757301e02cb2 (127.30.16.66:35723): tablet d66297e796a44c6e84b25b8975e82a80 (table test-table [id=86c798d4207c46edb0a2b0dbc067eaa5]) successfully deleted
I20260927 07:07:47.426967 9136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.384873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43988 (local address 127.30.16.65:38541)
0927 07:07:47.385047 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.385053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.407078 (+ 22025us) server_negotiation.cc:407] Connection header received
0927 07:07:47.407585 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.407593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.407676 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.407747 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:47.409664 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.410861 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.414713 (+ 3852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.415342 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.415476 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.420670 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:47.420690 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:47.420709 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:47.420762 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:47.423995 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.424899 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.424907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.424909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.425073 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.425488 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.425491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.425493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.425845 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:47.426101 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.426410 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.426638 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 07:07:47.427651 8849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:47.543303 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8726
I20260927 07:07:47.562026 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8860
Sep 27 07:07:47 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:07:47.579344 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.562870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.30.16.66:35723)
0927 07:07:47.563049 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.563056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.563107 (+ 51us) server_negotiation.cc:407] Connection header received
0927 07:07:47.563328 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.563334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.563408 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.563541 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:47.564806 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.565640 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.566981 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.567433 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.567588 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.572092 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:47.572118 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:47.572145 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:47.572199 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:47.576307 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.577138 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.577148 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.577153 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.577336 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.577712 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.577718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.577727 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.578161 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:47.578569 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.578730 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.578948 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:07:47.580176 8983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:47.695096 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8860
I20260927 07:07:47.714109 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8994
Sep 27 07:07:47 dist-test-slave-8vqf krb5kdc[8620](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:07:47.731290 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:07:47.715139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.30.16.67:40167)
0927 07:07:47.715340 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 07:07:47.715348 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:07:47.715371 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:07:47.715476 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:07:47.715484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:07:47.715577 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:07:47.715695 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:07:47.717078 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.717861 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.719662 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:07:47.720119 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:07:47.720273 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:07:47.725121 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:07:47.725139 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:07:47.725156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:07:47.725198 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:07:47.728217 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.728984 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.728993 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.728996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.729156 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:07:47.729584 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:07:47.729597 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:07:47.729600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:07:47.729990 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:07:47.730338 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0927 07:07:47.730693 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 07:07:47.730935 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:07:47.731993 9117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:47.826666 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8994
I20260927 07:07:47.843474 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 8633
I20260927 07:07:47.844640 8695 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:07:47.954290 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 8633
2026-09-27T07:07:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4650 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:07:48 dist-test-slave-8vqf krb5kdc[9174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:07:48 dist-test-slave-8vqf krb5kdc[9174](info): set up 2 sockets
Sep 27 07:07:48 dist-test-slave-8vqf krb5kdc[9174](info): commencing operation
krb5kdc: starting...
W20260927 07:07:49.143666 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.143s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:07:49 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:07:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:07:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangeradmin_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangeradmin_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangerlookup_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangerlookup_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
I20260927 07:07:49.482304 30785 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhkCnwT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/postgres -l logfile start
2026-09-27 07:07:52.172 UTC [9204] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:07:52.172 UTC [9204] LOG: listening on IPv4 address "127.30.16.84", port 57247
2026-09-27 07:07:52.176 UTC [9204] LOG: listening on Unix socket "/tmp/.s.PGSQL.57247"
2026-09-27 07:07:52.201 UTC [9207] LOG: database system was shut down at 2026-09-27 07:07:50 UTC
2026-09-27 07:07:52.205 UTC [9204] LOG: database system is ready to accept connections
I20260927 07:07:53.179245 30785 mini_postgres.cc:96] Postgres bound to 57247
2026-09-27 07:07:53.198 UTC [9215] FATAL: database "slave" does not exist
127.30.16.84:57247 - accepting connections
I20260927 07:07:53.199369 30785 mini_ranger.cc:161] Starting Ranger...
I20260927 07:07:53.226328 30785 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:07:53.796726 30785 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:07:53.796851 30785 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin
2026-09-27 07:07:54,108 [I] DB FLAVOR :POSTGRES
2026-09-27 07:07:54,108 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:07:54,108 [I] Checking connection..
2026-09-27 07:07:54,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:07:54,355 [I] Checking connection passed.
2026-09-27 07:07:54,355 [I] --------- Verifying version history table ---------
2026-09-27 07:07:54,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:07:54,622 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:07:54,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:07:54,860 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:07:54,861 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:07:54,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:07:55.116 UTC [9279] WARNING: there is no transaction in progress
2026-09-27 07:07:55,122 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:07:55,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:07:55,388 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:07:55,389 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:07:55,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:07:55,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:07:55,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vqf.c.gcp-upstream.internal','N') ;"
2026-09-27 07:07:56,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:07:56,408 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:07:56,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:07:56,662 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:07:56,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:07:56,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:07:57,131 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:07:57,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:07:58.183 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.201 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.219 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.235 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.252 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.270 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.331 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.338 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.355 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.366 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.378 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.394 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.404 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.411 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.419 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:58.427 UTC [9387] WARNING: there is no transaction in progress
2026-09-27 07:07:59,140 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:07:59,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:07:59,378 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:07:59,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:07:59,622 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:07:59,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:07:59,891 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:07:59,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:08:00,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:08:00,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vqf.c.gcp-upstream.internal';"
2026-09-27 07:08:00,633 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:08:00,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:08:00,885 [I] DB_PATCHES have already been applied
I20260927 07:08:00.891490 30785 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhkCnwT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:08:00.891585 30785 mini_ranger.cc:193] Using host: 127.30.16.84
I20260927 07:08:00.907105 30785 mini_ranger.cc:227] Ranger admin URL: http://127.30.16.84:48273
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:08:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:08:02.082760 30785 mini_ranger.cc:160] Time spent starting Ranger: real 8.883s user 0.016s sys 0.028s
Sep 27, 2026 7:08:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:08:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 7:08:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:08:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:08:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:08:24.186470 30785 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangerkms_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/rangerkms_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.16.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.16.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:08:24 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:08:24.351233 30785 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:08:24.373349 30785 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:08:25.332000 30785 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:08:25.332168 30785 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms
I20260927 07:08:25.332322 30785 mini_ranger_kms.cc:227] Using postgres at 127.30.16.84:57247
2026-09-27 07:08:25,752 [I] DB FLAVOR :POSTGRES
2026-09-27 07:08:25,753 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:08:25,753 [I] Checking connection
2026-09-27 07:08:25,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:08:26,002 [I] connection success
2026-09-27 07:08:26,002 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:08:26,002 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:08:26,003 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:08:26,248 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:08:26,249 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:08:26,249 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:08:26,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57247/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:08:26,519 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:08:26.983217 30785 mini_ranger_kms.cc:310] Created kms service
I20260927 07:08:27.161196 30785 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:08:27.212430 30785 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:08:27 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:27.653285 30785 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:08:27.653599 30785 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:08:27.653646 30785 mini_ranger_kms.cc:240] Using host: 127.30.16.84
I20260927 07:08:27.673113 30785 mini_ranger_kms.cc:276] Ranger KMS PID: 9568
I20260927 07:08:27.673437 30785 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.16.84:59789
Sep 27, 2026 7:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:08:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:08:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 07:08:29 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:30.007766 30785 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.657s user 0.114s sys 0.329s
I20260927 07:08:30.008033 30785 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:08:30.008268 30785 mini_ranger_kms.cc:401] 127.30.16.84:59789/kms/v1/keys
Sep 27 07:08:30 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
Sep 27 07:08:30 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
Sep 27 07:08:33 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:08:33.242865 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34665
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:34665
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59789/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:08:33.590323 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:33.590704 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:33.590813 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:33.598317 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:08:33.598429 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:33.598475 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:33.598595 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:08:33.598650 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:08:33.609218 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.16.84:59789/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:34665
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:08:33.610539 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:33.611960 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:33.624826 9623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:33.624656 9626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:33.624653 9624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:33.624689 9618 server_base.cc:1034] running on GCE node
I20260927 07:08:33.626024 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:33.627633 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:33.628868 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790492913628795 us; error 71 us; skew 500 ppm
I20260927 07:08:33.629242 9618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:08:33 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:33.637970 9618 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:08:33.639636 9618 webserver.cc:460] Webserver started at http://127.30.16.126:41715/ using document root <none> and password file <none>
I20260927 07:08:33.640412 9618 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:33.640511 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:33.640767 9618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:08:33 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:33.701797 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "e3499ed5b40f40e3acf72b8560e8f14c"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-8vqf"
server_key: "37f1f49fac1b9891e269856e3592ff6d"
server_key_iv: "cc941ee63e2d5b6a5d879d19ae02047c"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:33.702827 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "e3499ed5b40f40e3acf72b8560e8f14c"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-8vqf"
server_key: "37f1f49fac1b9891e269856e3592ff6d"
server_key_iv: "cc941ee63e2d5b6a5d879d19ae02047c"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:33.709188 9618 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 07:08:33.731195 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:33.733294 9618 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260927 07:08:33.733517 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "e3499ed5b40f40e3acf72b8560e8f14c"
format_stamp: "Formatted at 2026-09-27 07:08:33 on dist-test-slave-8vqf"
server_key: "37f1f49fac1b9891e269856e3592ff6d"
server_key_iv: "cc941ee63e2d5b6a5d879d19ae02047c"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:33.733777 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:33.754971 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:33.759374 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:33.759702 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:33.778577 9618 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:34665
I20260927 07:08:33.778609 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:34665 every 8 connection(s)
I20260927 07:08:33.780046 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:08:33.781721 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 9618
I20260927 07:08:33.781932 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:08:33.782785 30785 external_mini_cluster.cc:1428] Setting key 1ddbdeb58631b2bbc843af441fb8d547
I20260927 07:08:33.785056 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:08:33 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:08:33.798532 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: Bootstrap starting.
I20260927 07:08:33.802012 9685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:33.803131 9685 log.cc:824] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:33.806051 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: No bootstrap required, opened a new log
I20260927 07:08:33.806522 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:33.784957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44956 (local address 127.30.16.126:34665)
0927 07:08:33.785782 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:33.785794 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:33.785836 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:08:33.787488 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:33.787515 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:33.787949 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:33.788292 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:33.789910 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.791429 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.793102 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:33.793790 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:33.794013 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:33.798703 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:33.798734 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:33.798758 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:33.798815 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:33.802300 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.803122 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.803131 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.803140 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.803338 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:33.803725 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:33.803733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:33.803737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:33.804303 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:33.804615 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:33.805053 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 07:08:33.805468 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":406,"threads_started":1}
I20260927 07:08:33.814193 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } }
I20260927 07:08:33.814608 9685 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:33.814720 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3499ed5b40f40e3acf72b8560e8f14c, State: Initialized, Role: FOLLOWER
I20260927 07:08:33.815271 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } }
I20260927 07:08:33.815505 9685 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:08:33.815649 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:08:33.815817 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:33.818303 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } }
I20260927 07:08:33.818914 9685 leader_election.cc:302] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3499ed5b40f40e3acf72b8560e8f14c; no voters:
I20260927 07:08:33.819664 9685 leader_election.cc:288] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:08:33.819833 9690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:33.820567 9690 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [term 1 LEADER]: Becoming Leader. State: Replica: e3499ed5b40f40e3acf72b8560e8f14c, State: Running, Role: LEADER
I20260927 07:08:33.821167 9690 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } }
I20260927 07:08:33.821753 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:08:33.824813 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } } }
I20260927 07:08:33.825387 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:33.824895 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3499ed5b40f40e3acf72b8560e8f14c. Latest consensus state: current_term: 1 leader_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3499ed5b40f40e3acf72b8560e8f14c" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 34665 } } }
I20260927 07:08:33.826691 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c [sys.catalog]: This master's current role is: LEADER
I20260927 07:08:33.831941 9696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:08:33.837957 9696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:08:33.838841 9685 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:08:33.858188 9696 catalog_manager.cc:1347] Generated new cluster ID: 7d7fbe5680094a04bdfd00e736930882
I20260927 07:08:33.858474 9696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:08:33.893787 9696 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:08:33.895991 9696 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:08:33.915930 9696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: Generated new TSK 0
I20260927 07:08:33.917135 9696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:08:34.667811 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:08:34 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492914, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 07:08:35.130379 9730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:35.130957 9730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:35.131193 9730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:35.143282 9730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:35.143523 9730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:35.143783 9730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:08:35.160277 9730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.16.84:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:08:35.162078 9730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:35.164479 9730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:35.182264 9740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:35.182243 9741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:35.182644 9743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:35.184960 9730 server_base.cc:1034] running on GCE node
I20260927 07:08:35.185636 9730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:35.186895 9730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:35.188126 9730 hybrid_clock.cc:648] HybridClock initialized: now 1790492915188065 us; error 56 us; skew 500 ppm
I20260927 07:08:35.188485 9730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:08:35 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492915, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:35.200599 9730 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:08:35.211254 9730 webserver.cc:460] Webserver started at http://127.30.16.65:43147/ using document root <none> and password file <none>
I20260927 07:08:35.212126 9730 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:35.212283 9730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:35.212635 9730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:08:35 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492915, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:35.236991 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "d05548642a92410682ceaef34ccaad69"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-8vqf"
server_key: "63b82a90cba6b6a24a303e39e9bb65f7"
server_key_iv: "fc9e98f2fa9507557bb6453c5d959629"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:35.239382 9730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "d05548642a92410682ceaef34ccaad69"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-8vqf"
server_key: "63b82a90cba6b6a24a303e39e9bb65f7"
server_key_iv: "fc9e98f2fa9507557bb6453c5d959629"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:35.250020 9730 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.006s sys 0.001s
I20260927 07:08:35.275743 9750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:35.278139 9730 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260927 07:08:35.278395 9730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "d05548642a92410682ceaef34ccaad69"
format_stamp: "Formatted at 2026-09-27 07:08:35 on dist-test-slave-8vqf"
server_key: "63b82a90cba6b6a24a303e39e9bb65f7"
server_key_iv: "fc9e98f2fa9507557bb6453c5d959629"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:35.278883 9730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:35.345564 9730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:35.350023 9730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:35.350358 9730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:35.351531 9730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:35.353081 9730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:35.353189 9730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:35.353328 9730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:35.353421 9730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:08:35 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492914, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:35.535679 9730 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:33223
I20260927 07:08:35.535707 9862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:33223 every 8 connection(s)
I20260927 07:08:35.537735 9730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:08:35.539026 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 9730
I20260927 07:08:35.539242 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:08:35.540068 30785 external_mini_cluster.cc:1428] Setting key 499200bae18c9c88601a1413c3914fdd
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Sep 27 07:08:35 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492915, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:08:35.596150 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:35.558999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:51517 (local address 127.30.16.126:34665)
0927 07:08:35.566981 (+ 7982us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:35.566989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:35.569782 (+ 2793us) server_negotiation.cc:407] Connection header received
0927 07:08:35.573112 (+ 3330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:35.573119 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:35.573237 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:35.573333 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:35.575832 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:35.576773 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:35.578629 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:35.579387 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:35.579541 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:35.588986 (+ 9445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:35.588994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:35.589001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:35.589058 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:35.592784 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:35.593600 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:35.593606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:35.593610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:35.593768 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:35.594235 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:35.594240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:35.594243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:35.594628 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:35.594771 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:35.595511 (+ 740us) server_negotiation.cc:299] Negotiation successful
0927 07:08:35.595750 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7829,"thread_start_us":7696,"threads_started":1}
I20260927 07:08:35.597289 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:35.558472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34665 (local address 127.30.16.65:51517)
0927 07:08:35.569582 (+ 11110us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:35.569641 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:35.572593 (+ 2952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:35.574010 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:35.574032 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:35.574720 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:35.575628 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:35.575649 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:35.576966 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:35.576973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:35.578446 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:35.578465 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:35.584153 (+ 5688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:35.584163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:35.584568 (+ 405us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:35.585528 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:35.585568 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:35.588684 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:35.592950 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:35.592969 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:35.592985 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:35.593447 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:35.593886 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:35.593897 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:35.593901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:35.594107 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:35.594747 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:35.594756 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:35.595138 (+ 382us) client_negotiation.cc:769] Sending connection context
0927 07:08:35.595420 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 07:08:35.595996 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10778,"thread_start_us":10445,"threads_started":1}
I20260927 07:08:35.599182 9863 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34665
I20260927 07:08:35.599514 9863 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:35.600282 9863 heartbeater.cc:511] Master 127.30.16.126:34665 requested a full tablet report, sending...
I20260927 07:08:35.602394 9650 ts_manager.cc:194] Registered new tserver with Master: d05548642a92410682ceaef34ccaad69 (127.30.16.65:33223)
I20260927 07:08:35.603914 9650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:51517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:08:35.615773 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:08:35 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492914, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
W20260927 07:08:36.125785 9869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:36.126314 9869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:36.126504 9869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:36.136363 9869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:36.136528 9869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:36.136711 9869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:08:36.147214 9869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.16.84:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:08:36.148669 9869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:36.150044 9869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:36.168900 9884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:36.175132 9883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:36.175283 9869 server_base.cc:1034] running on GCE node
W20260927 07:08:36.175643 9886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:36.176481 9869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:36.177781 9869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:36.179072 9869 hybrid_clock.cc:648] HybridClock initialized: now 1790492916179002 us; error 67 us; skew 500 ppm
I20260927 07:08:36.179432 9869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:36.182972 9869 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:08:36.185009 9869 webserver.cc:460] Webserver started at http://127.30.16.66:38749/ using document root <none> and password file <none>
I20260927 07:08:36.185870 9869 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:36.186017 9869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:36.186355 9869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:36.202445 9869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "e0acfd72bc39483a9daa2801337849c6"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "7ca350220ac1fca7ce35cb54e3b9d719"
server_key_iv: "6bf12d995b4c8955330a4bf49cdd3a14"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.203696 9869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "e0acfd72bc39483a9daa2801337849c6"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "7ca350220ac1fca7ce35cb54e3b9d719"
server_key_iv: "6bf12d995b4c8955330a4bf49cdd3a14"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.217728 9869 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260927 07:08:36.247275 9893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.249446 9869 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.010s
I20260927 07:08:36.249765 9869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "e0acfd72bc39483a9daa2801337849c6"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "7ca350220ac1fca7ce35cb54e3b9d719"
server_key_iv: "6bf12d995b4c8955330a4bf49cdd3a14"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.250113 9869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:36.291596 9869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:36.296798 9869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:36.297130 9869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:36.298290 9869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:36.300184 9869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:36.300275 9869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.300444 9869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:36.300518 9869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.336495 9869 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:45921
I20260927 07:08:36.336521 10005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:45921 every 8 connection(s)
I20260927 07:08:36.338449 9869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:08:36.340342 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 9869
I20260927 07:08:36.340531 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:08:36.341027 30785 external_mini_cluster.cc:1428] Setting key 56897a0820ebd68de41fe17ec993fd33
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:08:36.386620 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:36.344327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34665 (local address 127.30.16.66:38535)
0927 07:08:36.363407 (+ 19080us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:36.363487 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:36.366582 (+ 3095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:36.368097 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:36.368107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:36.368585 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:36.369220 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:36.369238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.370744 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.370749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:36.372074 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:36.372094 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.373302 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.373309 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:36.373614 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:36.374464 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:36.374494 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:36.377306 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:36.381360 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:36.381371 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:36.381392 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:36.382018 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:36.382468 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:36.382473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:36.382477 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:36.382785 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:36.383506 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:36.383516 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:36.384036 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 07:08:36.384325 (+ 289us) client_negotiation.cc:240] Negotiation successful
0927 07:08:36.384880 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18642,"thread_start_us":18351,"threads_started":1}
I20260927 07:08:36.387883 10008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:36.343684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:38535 (local address 127.30.16.126:34665)
0927 07:08:36.367471 (+ 23787us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:36.367477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:36.367500 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:08:36.367580 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:36.367584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:36.367679 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:36.367801 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:36.369428 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.370595 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.372240 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.373048 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.374057 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:36.377604 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:36.377610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:36.377615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:36.377656 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:36.381176 (+ 3520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:36.382180 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:36.382185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:36.382189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:36.382334 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:36.382939 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:36.382944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:36.382948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:36.383350 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:36.386713 (+ 3363us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:36.387175 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 07:08:36.387528 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23647,"thread_start_us":23956,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:08:36.389293 10006 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34665
I20260927 07:08:36.389629 10006 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:36.390443 10006 heartbeater.cc:511] Master 127.30.16.126:34665 requested a full tablet report, sending...
I20260927 07:08:36.395071 9650 ts_manager.cc:194] Registered new tserver with Master: e0acfd72bc39483a9daa2801337849c6 (127.30.16.66:45921)
I20260927 07:08:36.396222 9650 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:38535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:08:36.423234 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59789/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:43047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:08:36.607170 9863 heartbeater.cc:503] Master 127.30.16.126:34665 was elected leader, sending a full tablet report...
W20260927 07:08:36.817816 10012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:08:36.818169 10012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:08:36.818336 10012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:08:36.825943 10012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:08:36.826109 10012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:08:36.826331 10012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:08:36.836690 10012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:43047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.16.84:59789/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:34665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:08:36.838182 10012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:08:36.839681 10012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:08:36.853127 10017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:36.853257 10018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:08:36.853186 10020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:08:36.854734 10012 server_base.cc:1034] running on GCE node
I20260927 07:08:36.855494 10012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:08:36.856638 10012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:08:36.857880 10012 hybrid_clock.cc:648] HybridClock initialized: now 1790492916857792 us; error 86 us; skew 500 ppm
I20260927 07:08:36.858268 10012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:08:36.861733 10012 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:08:36.863410 10012 webserver.cc:460] Webserver started at http://127.30.16.67:43421/ using document root <none> and password file <none>
I20260927 07:08:36.864154 10012 fs_manager.cc:359] Metadata directory not provided
I20260927 07:08:36.864281 10012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:08:36.864599 10012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:08:36.879957 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "787033785c5a4accbab2c2810528f294"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "39a4c182a1da5b37cedfce4fca977f95"
server_key_iv: "d83e4bf359d649c8d9d04f4a7242ac15"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.880831 10012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "787033785c5a4accbab2c2810528f294"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "39a4c182a1da5b37cedfce4fca977f95"
server_key_iv: "d83e4bf359d649c8d9d04f4a7242ac15"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.886725 10012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 07:08:36.903389 10027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.905056 10012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:08:36.905260 10012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "787033785c5a4accbab2c2810528f294"
format_stamp: "Formatted at 2026-09-27 07:08:36 on dist-test-slave-8vqf"
server_key: "39a4c182a1da5b37cedfce4fca977f95"
server_key_iv: "d83e4bf359d649c8d9d04f4a7242ac15"
server_key_version: "kuduclusterkey@0"
I20260927 07:08:36.905506 10012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:08:36.926029 10012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:08:36.929936 10012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:08:36.930259 10012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:08:36.931303 10012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:08:36.932794 10012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:08:36.932888 10012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.933053 10012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:08:36.933125 10012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:08:36.964748 10012 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36997
I20260927 07:08:36.964758 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36997 every 8 connection(s)
I20260927 07:08:36.966060 10012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:08:36.975113 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 10012
I20260927 07:08:36.975287 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:08:36.975749 30785 external_mini_cluster.cc:1428] Setting key 138eeba88bf0711de4f5e465e0bd55bf
Sep 27 07:08:36 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492916, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:08:36.987833 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:36.968815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:33825 (local address 127.30.16.126:34665)
0927 07:08:36.969321 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:36.969326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:36.970424 (+ 1098us) server_negotiation.cc:407] Connection header received
0927 07:08:36.972500 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:36.972507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:36.972605 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:36.972731 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:36.974406 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.975371 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.977288 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.977926 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.978059 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:36.981429 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:36.981435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:36.981438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:36.981472 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:36.984441 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:36.985299 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:36.985304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:36.985306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:36.985444 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:36.986118 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:36.986121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:36.986124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:36.986411 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:36.986617 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:36.987312 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 07:08:36.987511 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":232,"threads_started":1}
I20260927 07:08:36.988876 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:36.969417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:34665 (local address 127.30.16.67:33825)
0927 07:08:36.970236 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:36.970284 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:36.972175 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:36.972999 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:36.973009 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:36.973464 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:08:36.974224 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:36.974244 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.975551 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.975555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:36.977121 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:36.977141 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:36.978124 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:36.978128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:36.978410 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:36.979186 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:08:36.979208 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:08:36.981224 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:08:36.984589 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:36.984596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:36.984608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:36.985150 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:36.985644 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:08:36.985648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:08:36.985650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:08:36.986003 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:08:36.986535 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:08:36.986541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:08:36.986967 (+ 426us) client_negotiation.cc:769] Sending connection context
0927 07:08:36.987231 (+ 264us) client_negotiation.cc:240] Negotiation successful
0927 07:08:36.987638 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
I20260927 07:08:36.990563 10140 heartbeater.cc:347] Connected to a master server at 127.30.16.126:34665
I20260927 07:08:36.990945 10140 heartbeater.cc:464] Registering TS with master...
I20260927 07:08:36.991894 10140 heartbeater.cc:511] Master 127.30.16.126:34665 requested a full tablet report, sending...
I20260927 07:08:36.993429 9649 ts_manager.cc:194] Registered new tserver with Master: 787033785c5a4accbab2c2810528f294 (127.30.16.67:36997)
I20260927 07:08:36.994347 9649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:33825
I20260927 07:08:37.004413 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:08:37.022328 10142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.009152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.30.16.126:34665)
0927 07:08:37.009383 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.009388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.009842 (+ 454us) server_negotiation.cc:407] Connection header received
0927 07:08:37.010015 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.010020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.010152 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.010267 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.011835 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.012585 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.013929 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.014364 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.014467 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.017021 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.017026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.017029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.017059 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.019794 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.020425 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.020429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.020432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.020549 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.020898 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.020901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.020903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.021202 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.021370 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.021757 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.022097 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260927 07:08:37.026762 9649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:08:37.033981 9649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:08:37.058858 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.047096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:33223 (local address 127.0.0.1:44796)
0927 07:08:37.048015 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:37.048067 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:37.048243 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:37.051604 (+ 3361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:37.051608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:37.051629 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:37.052054 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.052066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.054071 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.054076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.056363 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.056377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.057760 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.057766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.057995 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.058050 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:08:37.058207 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 07:08:37.058400 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":1,"thread_start_us":2177,"threads_started":1}
I20260927 07:08:37.058962 10154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.047294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:45921 (local address 127.0.0.1:41202)
0927 07:08:37.048853 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:37.048873 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:37.048993 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:37.050665 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:37.050676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:37.050718 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:37.051343 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.051360 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.053378 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.053383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.056105 (+ 2722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.056119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.058183 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.058191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.058346 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.058375 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 07:08:37.058458 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 07:08:37.058641 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1435,"thread_start_us":1295,"threads_started":1}
I20260927 07:08:37.059831 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.047667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44796 (local address 127.30.16.65:33223)
0927 07:08:37.050159 (+ 2492us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.050169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.050196 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:08:37.050295 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.050307 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.050648 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.052172 (+ 1524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:37.052492 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.053854 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.056530 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.057530 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.057696 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.057947 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.059276 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.059486 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2325,"thread_start_us":2213,"threads_started":1}
I20260927 07:08:37.060271 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.048765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36997 (local address 127.0.0.1:37256)
0927 07:08:37.051316 (+ 2551us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:37.051339 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:37.051443 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:37.053286 (+ 1843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:37.053293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:37.053324 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:37.053810 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.053821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.055539 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.055544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.057712 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.057726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.059509 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.059514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.059690 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.059721 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:08:37.059781 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 07:08:37.059969 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2409,"mutex_wait_us":4,"thread_start_us":2275,"threads_started":1}
I20260927 07:08:37.060281 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.048082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41202 (local address 127.30.16.66:45921)
0927 07:08:37.049850 (+ 1768us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.049858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.049885 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:08:37.049984 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.049993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.050303 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.051569 (+ 1266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:37.051876 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.053191 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.056308 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.057884 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.058017 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.058201 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.058536 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.059948 (+ 1412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1601,"thread_start_us":1621,"threads_started":1}
I20260927 07:08:37.061367 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.049336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.30.16.67:36997)
0927 07:08:37.052690 (+ 3354us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.052701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.052734 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:08:37.052820 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.052828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.053092 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.053269 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:37.054013 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.055376 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.057893 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.059276 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.060620 (+ 1344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.060781 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.060832 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.061015 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3178,"thread_start_us":3011,"threads_started":1}
I20260927 07:08:37.063947 9941 tablet_service.cc:1467] Processing CreateTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 (DEFAULT_TABLE table=test-table [id=c7bf6f15a43e4918a6447112c7b5017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:37.063855 9798 tablet_service.cc:1467] Processing CreateTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 (DEFAULT_TABLE table=test-table [id=c7bf6f15a43e4918a6447112c7b5017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:37.064697 10075 tablet_service.cc:1467] Processing CreateTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 (DEFAULT_TABLE table=test-table [id=c7bf6f15a43e4918a6447112c7b5017b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:08:37.065402 9941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40565e7e7dda4e589f357a9b82b285c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:37.065760 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40565e7e7dda4e589f357a9b82b285c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:37.066035 10075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40565e7e7dda4e589f357a9b82b285c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:08:37.079722 10159 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Bootstrap starting.
I20260927 07:08:37.083326 10160 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Bootstrap starting.
I20260927 07:08:37.083662 10159 tablet_bootstrap.cc:654] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:37.084291 10161 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Bootstrap starting.
I20260927 07:08:37.085212 10159 log.cc:824] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:37.088261 10161 tablet_bootstrap.cc:654] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:37.088814 10160 tablet_bootstrap.cc:654] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Neither blocks nor log segments found. Creating new log.
I20260927 07:08:37.088903 10159 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: No bootstrap required, opened a new log
I20260927 07:08:37.089234 10159 ts_tablet_manager.cc:1397] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 07:08:37.089514 10161 log.cc:824] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:37.090047 10160 log.cc:824] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:08:37.092433 10161 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: No bootstrap required, opened a new log
I20260927 07:08:37.092694 10161 ts_tablet_manager.cc:1397] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 07:08:37.093159 10160 tablet_bootstrap.cc:492] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: No bootstrap required, opened a new log
I20260927 07:08:37.093420 10160 ts_tablet_manager.cc:1397] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260927 07:08:37.099987 10159 raft_consensus.cc:348] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.100226 10161 raft_consensus.cc:348] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.100510 10159 raft_consensus.cc:374] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:37.100621 10161 raft_consensus.cc:374] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:37.100642 10159 raft_consensus.cc:729] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05548642a92410682ceaef34ccaad69, State: Initialized, Role: FOLLOWER
I20260927 07:08:37.100744 10161 raft_consensus.cc:729] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787033785c5a4accbab2c2810528f294, State: Initialized, Role: FOLLOWER
I20260927 07:08:37.101511 10161 consensus_queue.cc:260] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.101531 10159 consensus_queue.cc:260] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.102025 10165 raft_consensus.cc:482] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:08:37.102408 10165 raft_consensus.cc:504] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.104640 10160 raft_consensus.cc:348] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.105266 10160 raft_consensus.cc:374] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:08:37.105412 10160 raft_consensus.cc:729] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0acfd72bc39483a9daa2801337849c6, State: Initialized, Role: FOLLOWER
I20260927 07:08:37.106367 10160 consensus_queue.cc:260] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.110117 10161 ts_tablet_manager.cc:1428] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
I20260927 07:08:37.110208 10159 ts_tablet_manager.cc:1428] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Time spent starting tablet: real 0.021s user 0.021s sys 0.000s
I20260927 07:08:37.110661 10140 heartbeater.cc:503] Master 127.30.16.126:34665 was elected leader, sending a full tablet report...
I20260927 07:08:37.112543 10006 heartbeater.cc:503] Master 127.30.16.126:34665 was elected leader, sending a full tablet report...
I20260927 07:08:37.113366 10165 leader_election.cc:288] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0acfd72bc39483a9daa2801337849c6 (127.30.16.66:45921), 787033785c5a4accbab2c2810528f294 (127.30.16.67:36997)
I20260927 07:08:37.113480 10160 ts_tablet_manager.cc:1428] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Time spent starting tablet: real 0.020s user 0.011s sys 0.008s
W20260927 07:08:37.119259 10007 tablet.cc:2369] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:08:37.122285 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.114152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:35021 (local address 127.30.16.67:36997)
0927 07:08:37.114388 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.114397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.115832 (+ 1435us) server_negotiation.cc:407] Connection header received
0927 07:08:37.116082 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.116087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.116163 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.116219 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:37.117866 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.118821 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.120560 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.121314 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.121524 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.121600 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.121882 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.122005 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:08:37.122386 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.114200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36997 (local address 127.30.16.65:35021)
0927 07:08:37.115745 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:37.115762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:37.115919 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:37.116421 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:37.116425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:37.116452 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:37.116939 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.116962 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.118994 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.118998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.120397 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.120409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.121561 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.121567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.121724 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.121748 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 07:08:37.121861 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 07:08:37.122005 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"thread_start_us":1276,"threads_started":1}
I20260927 07:08:37.123373 10095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40565e7e7dda4e589f357a9b82b285c4" candidate_uuid: "d05548642a92410682ceaef34ccaad69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787033785c5a4accbab2c2810528f294" is_pre_election: true
I20260927 07:08:37.123648 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.106234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:37559 (local address 127.30.16.66:45921)
0927 07:08:37.106494 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.106502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.115703 (+ 9201us) server_negotiation.cc:407] Connection header received
0927 07:08:37.115785 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.115793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.115885 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.115975 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:08:37.117830 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.118943 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.121553 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.122629 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.122776 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.122825 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.123282 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.123403 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:08:37.123680 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.105605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:45921 (local address 127.30.16.65:37559)
0927 07:08:37.115061 (+ 9456us) negotiation.cc:107] Waiting for socket to connect
0927 07:08:37.115094 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:08:37.115240 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:08:37.117216 (+ 1976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:08:37.117221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:08:37.117244 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:08:37.117635 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.117647 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.120069 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.120072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.121388 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:08:37.121399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.122945 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.122957 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:08:37.123070 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.123092 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:08:37.123208 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 07:08:37.123355 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9305,"thread_start_us":15934,"threads_started":1}
I20260927 07:08:37.123952 10095 raft_consensus.cc:2454] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d05548642a92410682ceaef34ccaad69 in term 0.
I20260927 07:08:37.124634 9751 leader_election.cc:302] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 787033785c5a4accbab2c2810528f294, d05548642a92410682ceaef34ccaad69; no voters:
I20260927 07:08:37.124711 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40565e7e7dda4e589f357a9b82b285c4" candidate_uuid: "d05548642a92410682ceaef34ccaad69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0acfd72bc39483a9daa2801337849c6" is_pre_election: true
I20260927 07:08:37.125020 10165 raft_consensus.cc:2781] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:08:37.125164 10165 raft_consensus.cc:482] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:08:37.125277 10165 raft_consensus.cc:3037] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:37.125279 9961 raft_consensus.cc:2454] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d05548642a92410682ceaef34ccaad69 in term 0.
I20260927 07:08:37.127781 10165 raft_consensus.cc:504] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.128489 10165 leader_election.cc:288] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [CANDIDATE]: Term 1 election: Requested vote from peers e0acfd72bc39483a9daa2801337849c6 (127.30.16.66:45921), 787033785c5a4accbab2c2810528f294 (127.30.16.67:36997)
I20260927 07:08:37.128858 9961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40565e7e7dda4e589f357a9b82b285c4" candidate_uuid: "d05548642a92410682ceaef34ccaad69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0acfd72bc39483a9daa2801337849c6"
I20260927 07:08:37.128960 10095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40565e7e7dda4e589f357a9b82b285c4" candidate_uuid: "d05548642a92410682ceaef34ccaad69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "787033785c5a4accbab2c2810528f294"
I20260927 07:08:37.129086 9961 raft_consensus.cc:3037] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:37.129156 10095 raft_consensus.cc:3037] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:08:37.132380 9961 raft_consensus.cc:2454] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d05548642a92410682ceaef34ccaad69 in term 1.
I20260927 07:08:37.132380 10095 raft_consensus.cc:2454] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d05548642a92410682ceaef34ccaad69 in term 1.
I20260927 07:08:37.132812 9752 leader_election.cc:302] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d05548642a92410682ceaef34ccaad69, e0acfd72bc39483a9daa2801337849c6; no voters:
I20260927 07:08:37.133222 10165 raft_consensus.cc:2781] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:08:37.133378 10165 raft_consensus.cc:686] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 1 LEADER]: Becoming Leader. State: Replica: d05548642a92410682ceaef34ccaad69, State: Running, Role: LEADER
I20260927 07:08:37.133841 10165 consensus_queue.cc:237] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } }
I20260927 07:08:37.138959 9650 catalog_manager.cc:5495] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 reported cstate change: term changed from 0 to 1, leader changed from <none> to d05548642a92410682ceaef34ccaad69 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "d05548642a92410682ceaef34ccaad69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d05548642a92410682ceaef34ccaad69" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 } health_report { overall_health: UNKNOWN } } }
I20260927 07:08:37.160408 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.154646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.30.16.65:33223)
0927 07:08:37.154866 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.154872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.154957 (+ 85us) server_negotiation.cc:407] Connection header received
0927 07:08:37.155102 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.155106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.155216 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.155397 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:08:37.155783 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.156598 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.158069 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.158470 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.158630 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.159152 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:08:37.159662 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:08:37.159861 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.160006 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.160094 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:08:37.168980 10095 raft_consensus.cc:1261] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 1 FOLLOWER]: Refusing update from remote peer d05548642a92410682ceaef34ccaad69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:37.169003 9961 raft_consensus.cc:1261] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 1 FOLLOWER]: Refusing update from remote peer d05548642a92410682ceaef34ccaad69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:08:37.170023 10167 consensus_queue.cc:1035] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0acfd72bc39483a9daa2801337849c6" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:37.170437 10165 consensus_queue.cc:1035] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "787033785c5a4accbab2c2810528f294" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:08:37.200469 9648 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:37.200742 9648 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44964:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:08:37.204464 9648 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3499ed5b40f40e3acf72b8560e8f14c: Sending DeleteTablet for 3 replicas of tablet 40565e7e7dda4e589f357a9b82b285c4
I20260927 07:08:37.205649 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:37 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:44796
I20260927 07:08:37.205729 10075 tablet_service.cc:1514] Processing DeleteTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:37 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:37256
I20260927 07:08:37.205708 9941 tablet_service.cc:1514] Processing DeleteTablet for tablet 40565e7e7dda4e589f357a9b82b285c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:08:37 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:41202
I20260927 07:08:37.209988 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 9730
I20260927 07:08:37.216032 10184 tablet_replica.cc:330] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: stopping tablet replica
I20260927 07:08:37.217167 10184 raft_consensus.cc:2229] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:37.218452 10184 raft_consensus.cc:2258] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:37.221441 10185 tablet_replica.cc:330] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: stopping tablet replica
I20260927 07:08:37.221693 10183 tablet_replica.cc:330] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: stopping tablet replica
I20260927 07:08:37.221686 10184 ts_tablet_manager.cc:1905] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:37.221917 10185 raft_consensus.cc:2229] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:08:37.222174 10183 raft_consensus.cc:2229] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:08:37.222421 10185 raft_consensus.cc:2258] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:37.223049 10183 raft_consensus.cc:2258] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:08:37.225328 10183 ts_tablet_manager.cc:1905] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:08:37.225867 10185 ts_tablet_manager.cc:1905] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 07:08:37 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:08:37.233144 10185 ts_tablet_manager.cc:1918] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:37.233357 10185 log.cc:1196] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/40565e7e7dda4e589f357a9b82b285c4
I20260927 07:08:37.233937 10185 ts_tablet_manager.cc:1939] T 40565e7e7dda4e589f357a9b82b285c4 P e0acfd72bc39483a9daa2801337849c6: Deleting consensus metadata
I20260927 07:08:37.235569 10184 ts_tablet_manager.cc:1918] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:37.236151 9635 catalog_manager.cc:4841] TS e0acfd72bc39483a9daa2801337849c6 (127.30.16.66:45921): tablet 40565e7e7dda4e589f357a9b82b285c4 (table test-table [id=c7bf6f15a43e4918a6447112c7b5017b]) successfully deleted
I20260927 07:08:37.236717 10184 log.cc:1196] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/40565e7e7dda4e589f357a9b82b285c4
I20260927 07:08:37.237362 10184 ts_tablet_manager.cc:1939] T 40565e7e7dda4e589f357a9b82b285c4 P 787033785c5a4accbab2c2810528f294: Deleting consensus metadata
I20260927 07:08:37.239516 9634 catalog_manager.cc:4841] TS 787033785c5a4accbab2c2810528f294 (127.30.16.67:36997): tablet 40565e7e7dda4e589f357a9b82b285c4 (table test-table [id=c7bf6f15a43e4918a6447112c7b5017b]) successfully deleted
I20260927 07:08:37.239917 10183 ts_tablet_manager.cc:1918] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:08:37.240108 10183 log.cc:1196] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/40565e7e7dda4e589f357a9b82b285c4
I20260927 07:08:37.240702 10183 ts_tablet_manager.cc:1939] T 40565e7e7dda4e589f357a9b82b285c4 P d05548642a92410682ceaef34ccaad69: Deleting consensus metadata
I20260927 07:08:37.240729 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.211419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.30.16.65:33223)
0927 07:08:37.211596 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.211604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.220171 (+ 8567us) server_negotiation.cc:407] Connection header received
0927 07:08:37.220497 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.220505 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.220598 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.220675 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.222477 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.223797 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.225762 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.226443 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.226539 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.232361 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.232377 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.232395 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.232470 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.236918 (+ 4448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.237881 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.237890 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.237894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.238087 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.238593 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.238600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.238608 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.239033 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.239347 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.239987 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.240276 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:08:37.242200 9858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:08:37.242669 9636 catalog_manager.cc:4841] TS d05548642a92410682ceaef34ccaad69 (127.30.16.65:33223): tablet 40565e7e7dda4e589f357a9b82b285c4 (table test-table [id=c7bf6f15a43e4918a6447112c7b5017b]) successfully deleted
I20260927 07:08:37.345256 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 9730
I20260927 07:08:37.369370 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 9869
Sep 27 07:08:37 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:08:37.387660 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.370212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.30.16.66:45921)
0927 07:08:37.370445 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.370451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.370469 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:08:37.370688 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.370695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.370771 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.370894 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.372143 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.373367 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.374838 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.375266 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.375376 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.380307 (+ 4931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.380321 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.380338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.380374 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.384251 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.385068 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.385081 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.385087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.385263 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.385807 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.385814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.385818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.386383 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.386722 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.387213 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.387396 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260927 07:08:37.388351 10001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:08:37.480266 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 9869
I20260927 07:08:37.497264 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 10012
Sep 27 07:08:37 dist-test-slave-8vqf krb5kdc[9174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:08:37.512367 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:08:37.498068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.30.16.67:36997)
0927 07:08:37.498256 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 07:08:37.498261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:08:37.498353 (+ 92us) server_negotiation.cc:407] Connection header received
0927 07:08:37.498500 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:08:37.498505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:08:37.498611 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:08:37.498703 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:08:37.500064 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.500794 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.502046 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:08:37.502431 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:08:37.502531 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:08:37.507025 (+ 4494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:08:37.507038 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:08:37.507054 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:08:37.507090 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:08:37.509882 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.510649 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.510656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.510658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.510774 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:08:37.511101 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:08:37.511105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:08:37.511107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:08:37.511462 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:08:37.511685 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 07:08:37.511933 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 07:08:37.512089 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:08:37.512984 10135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:08:37.603190 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 10012
I20260927 07:08:37.620362 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 9618
I20260927 07:08:37.621677 9680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:08:37.737092 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 9618
I20260927 07:08:37.759410 30785 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:08:38.095336 30785 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:08:38.095539 30785 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:08:38.441247 30785 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:08:38.441 UTC [9204] LOG: received smart shutdown request
2026-09-27 07:08:38.445 UTC [9204] LOG: background worker "logical replication launcher" (PID 9213) exited with exit code 1
2026-09-27 07:08:38.445 UTC [9208] LOG: shutting down
2026-09-27 07:08:38.652 UTC [9204] LOG: database system is shut down
2026-09-27T07:08:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50817 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:08:38 dist-test-slave-8vqf krb5kdc[10199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:08:38 dist-test-slave-8vqf krb5kdc[10199](info): set up 2 sockets
Sep 27 07:08:38 dist-test-slave-8vqf krb5kdc[10199](info): commencing operation
krb5kdc: starting...
W20260927 07:08:39.911103 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:08:40 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:08:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:08:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangeradmin_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangeradmin_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangerlookup_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangerlookup_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
I20260927 07:08:40.249938 30785 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhkCnwT/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/postgres -l logfile start
2026-09-27 07:08:43.038 UTC [10229] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 07:08:43.038 UTC [10229] LOG: listening on IPv4 address "127.30.16.84", port 57411
2026-09-27 07:08:43.042 UTC [10229] LOG: listening on Unix socket "/tmp/.s.PGSQL.57411"
2026-09-27 07:08:43.067 UTC [10232] LOG: database system was shut down at 2026-09-27 07:08:41 UTC
2026-09-27 07:08:43.071 UTC [10229] LOG: database system is ready to accept connections
I20260927 07:08:44.078778 30785 mini_postgres.cc:96] Postgres bound to 57411
2026-09-27 07:08:44.096 UTC [10240] FATAL: database "slave" does not exist
127.30.16.84:57411 - accepting connections
I20260927 07:08:44.097412 30785 mini_ranger.cc:161] Starting Ranger...
I20260927 07:08:44.124049 30785 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 07:08:44.677348 30785 mini_ranger.cc:87] Created ranger Postgres database
I20260927 07:08:44.677474 30785 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin
2026-09-27 07:08:44,982 [I] DB FLAVOR :POSTGRES
2026-09-27 07:08:44,983 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:08:44,983 [I] Checking connection..
2026-09-27 07:08:44,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 07:08:45,243 [I] Checking connection passed.
2026-09-27 07:08:45,243 [I] --------- Verifying version history table ---------
2026-09-27 07:08:45,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:08:45,506 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:08:45,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:08:45,772 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 07:08:45,772 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 07:08:45,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 07:08:46.014 UTC [10304] WARNING: there is no transaction in progress
2026-09-27 07:08:46,021 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 07:08:46,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 07:08:46,280 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 07:08:46,281 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:08:46,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 07:08:46,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 07:08:46,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-8vqf.c.gcp-upstream.internal','N') ;"
2026-09-27 07:08:47,032 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:08:47,297 [I] Table x_portal_user does not exist in database ranger
2026-09-27 07:08:47,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:08:47,555 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 07:08:47,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:08:47,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:08:48,062 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 07:08:48,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 07:08:49.086 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.105 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.123 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.139 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.156 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.176 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.235 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.243 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.259 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.270 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.281 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.296 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.307 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.314 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.322 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:49.330 UTC [10412] WARNING: there is no transaction in progress
2026-09-27 07:08:50,073 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 07:08:50,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 07:08:50,311 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 07:08:50,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 07:08:50,566 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 07:08:50,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 07:08:50,832 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 07:08:50,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 07:08:51,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 07:08:51,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-8vqf.c.gcp-upstream.internal';"
2026-09-27 07:08:51,634 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 07:08:51,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 07:08:51,901 [I] DB_PATCHES have already been applied
I20260927 07:08:51.907743 30785 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhkCnwT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 07:08:51.907837 30785 mini_ranger.cc:193] Using host: 127.30.16.84
I20260927 07:08:51.923030 30785 mini_ranger.cc:227] Ranger admin URL: http://127.30.16.84:41167
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 7:08:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 07:08:53.078034 30785 mini_ranger.cc:160] Time spent starting Ranger: real 8.981s user 0.004s sys 0.037s
Sep 27, 2026 7:09:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 7:09:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 7:09:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 7:09:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 7:09:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 07:09:14.134042 30785 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.30.16.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.30.16.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangerkms_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.30.16.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/rangerkms_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.30.16.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.30.16.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.30.16.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/HTTP_127.30.16.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 07:09:14 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 07:09:14.309142 30785 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 07:09:14.330734 30785 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 07:09:15.312358 30785 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 07:09:15.312484 30785 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms
I20260927 07:09:15.312531 30785 mini_ranger_kms.cc:227] Using postgres at 127.30.16.84:57411
2026-09-27 07:09:15,688 [I] DB FLAVOR :POSTGRES
2026-09-27 07:09:15,688 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 07:09:15,688 [I] Checking connection
2026-09-27 07:09:15,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 07:09:15,940 [I] connection success
2026-09-27 07:09:15,940 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 07:09:15,940 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 07:09:15,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 07:09:16,185 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 07:09:16,186 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 07:09:16,186 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 07:09:16,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.30.16.84:57411/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 07:09:16,453 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 07:09:16.930946 30785 mini_ranger_kms.cc:310] Created kms service
I20260927 07:09:17.138653 30785 mini_ranger_kms.cc:326] Created kudu user
I20260927 07:09:17.188131 30785 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 07:09:17 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492954, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:17.633049 30785 mini_ranger_kms.cc:384] Added ranger policy
I20260927 07:09:17.633409 30785 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhkCnwT/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhkCnwT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 07:09:17.633482 30785 mini_ranger_kms.cc:240] Using host: 127.30.16.84
I20260927 07:09:17.651163 30785 mini_ranger_kms.cc:276] Ranger KMS PID: 10592
I20260927 07:09:17.651306 30785 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.30.16.84:59847
Sep 27, 2026 7:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 7:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 7:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 7:09:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 7:09:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 7:09:19 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 07:09:19.933323 30785 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.624s user 0.055s sys 0.337s
I20260927 07:09:19.933578 30785 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 07:09:19.933749 30785 mini_ranger_kms.cc:401] 127.30.16.84:59847/kms/v1/keys
Sep 27 07:09:19 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 07:09:20 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
Sep 27 07:09:20 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492959, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.30.16.84@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
Sep 27 07:09:23 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:09:23.194896 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36417
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:36417
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:23.540750 10643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:23.541057 10643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:23.541227 10643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:23.548650 10643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:23.548766 10643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:23.548861 10643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:23.548942 10643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:23.549028 10643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:23.560032 10643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
--ranger_kms_url=127.30.16.84:59847/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:36417
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:23.561532 10643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:23.563005 10643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:23.575924 10648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:23.576023 10649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:23.576217 10643 server_base.cc:1034] running on GCE node
W20260927 07:09:23.575989 10651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:23.577453 10643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:23.579030 10643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:23.580260 10643 hybrid_clock.cc:648] HybridClock initialized: now 1790492963580219 us; error 37 us; skew 500 ppm
I20260927 07:09:23.580638 10643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:23 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:23.589349 10643 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:23.591086 10643 webserver.cc:460] Webserver started at http://127.30.16.126:43867/ using document root <none> and password file <none>
I20260927 07:09:23.591796 10643 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:23.591920 10643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:23.592248 10643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:09:23 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:23.657754 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "9c2bdd8d10134fb0be068587ff34ad65"
format_stamp: "Formatted at 2026-09-27 07:09:23 on dist-test-slave-8vqf"
server_key: "777e8ea35cce11dbe73b03b3f6b601f4"
server_key_iv: "2a53bde4b9965b9fe2aaeb289ff02721"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:23.658922 10643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "9c2bdd8d10134fb0be068587ff34ad65"
format_stamp: "Formatted at 2026-09-27 07:09:23 on dist-test-slave-8vqf"
server_key: "777e8ea35cce11dbe73b03b3f6b601f4"
server_key_iv: "2a53bde4b9965b9fe2aaeb289ff02721"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:23.666427 10643 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260927 07:09:23.683902 10658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:23.685575 10643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:09:23.685788 10643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "9c2bdd8d10134fb0be068587ff34ad65"
format_stamp: "Formatted at 2026-09-27 07:09:23 on dist-test-slave-8vqf"
server_key: "777e8ea35cce11dbe73b03b3f6b601f4"
server_key_iv: "2a53bde4b9965b9fe2aaeb289ff02721"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:23.686028 10643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:23.711493 10643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:23.715868 10643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:23.716212 10643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:23.736840 10643 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:36417
I20260927 07:09:23.736841 10709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:36417 every 8 connection(s)
I20260927 07:09:23.738332 10643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:23.742251 10710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:23.742373 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 10643
I20260927 07:09:23.742654 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:09:23.743566 30785 external_mini_cluster.cc:1428] Setting key 5d54a48976e43bf1cd112999dc9c2bde
I20260927 07:09:23.761375 10710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Bootstrap starting.
I20260927 07:09:23.765462 10710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:23.766839 10710 log.cc:824] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:23.772312 10710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: No bootstrap required, opened a new log
Sep 27 07:09:23 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:23.782594 10710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:23.783154 10710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:23.783299 10710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2bdd8d10134fb0be068587ff34ad65, State: Initialized, Role: FOLLOWER
I20260927 07:09:23.784015 10710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:23.784312 10710 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:23.784482 10710 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:23.784674 10710 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:23.785051 10713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:23.745736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.30.16.126:36417)
0927 07:09:23.759190 (+ 13454us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:23.759212 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:23.759683 (+ 471us) server_negotiation.cc:407] Connection header received
0927 07:09:23.762320 (+ 2637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:23.762346 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:23.762952 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:23.763480 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:23.765143 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:23.767256 (+ 2113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:23.769461 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:23.770312 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:23.770513 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:23.775519 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:23.775531 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:23.775550 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:23.775601 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:23.779777 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:23.780679 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:23.780689 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:23.780705 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:23.780911 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:23.781349 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:23.781355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:23.781358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:23.782096 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:23.782393 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:23.782917 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 07:09:23.783549 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13169,"thread_start_us":12892,"threads_started":1}
I20260927 07:09:23.787573 10710 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:23.788152 10710 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2bdd8d10134fb0be068587ff34ad65; no voters:
I20260927 07:09:23.788875 10710 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:09:23.789074 10715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:23.791231 10710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:23.795209 10715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2bdd8d10134fb0be068587ff34ad65, State: Running, Role: LEADER
I20260927 07:09:23.795922 10715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:23.798808 10716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2bdd8d10134fb0be068587ff34ad65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } } }
I20260927 07:09:23.799192 10716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:23.799120 10717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2bdd8d10134fb0be068587ff34ad65. Latest consensus state: current_term: 1 leader_uuid: "9c2bdd8d10134fb0be068587ff34ad65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } } }
I20260927 07:09:23.799427 10717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:23.799896 10710 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 07:09:23.800827 10729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:23.812839 10729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:23.825687 10729 catalog_manager.cc:1347] Generated new cluster ID: 1eef9c7d664e47c38197c0093b103ffb
I20260927 07:09:23.825874 10729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:23.842573 10729 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:09:23.844655 10729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:23.863523 10729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Generated new TSK 0
I20260927 07:09:23.864681 10729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:09:24.570833 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:24 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 07:09:24.986071 10755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:24.986490 10755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:24.986723 10755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:24.998409 10755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:24.998662 10755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:24.998871 10755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:09:25.015108 10755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:25.016896 10755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:25.018770 10755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:25.039402 10765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:25.040289 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:25.048884 10755 server_base.cc:1034] running on GCE node
W20260927 07:09:25.049229 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:25.050132 10755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:25.051465 10755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:25.052767 10755 hybrid_clock.cc:648] HybridClock initialized: now 1790492965052696 us; error 62 us; skew 500 ppm
I20260927 07:09:25.053172 10755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:25.057530 10755 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:09:25.059691 10755 webserver.cc:460] Webserver started at http://127.30.16.65:35049/ using document root <none> and password file <none>
I20260927 07:09:25.060708 10755 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:25.060850 10755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:25.061246 10755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:25.085659 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "e40308d95d3247459639d50109706a9e"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "192c1245238f82161c281cc075d5d8bc"
server_key_iv: "4fbcdc375426c54d8741ecac953284c7"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.088044 10755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "e40308d95d3247459639d50109706a9e"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "192c1245238f82161c281cc075d5d8bc"
server_key_iv: "4fbcdc375426c54d8741ecac953284c7"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.099797 10755 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.005s
I20260927 07:09:25.127856 10775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:25.129936 10755 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.004s sys 0.004s
I20260927 07:09:25.130146 10755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e40308d95d3247459639d50109706a9e"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "192c1245238f82161c281cc075d5d8bc"
server_key_iv: "4fbcdc375426c54d8741ecac953284c7"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.130425 10755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:25.154263 10755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:25.159916 10755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:25.160323 10755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:25.161670 10755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:25.163623 10755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:25.163718 10755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:25.163883 10755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:25.163961 10755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:25.243728 10755 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41131
I20260927 07:09:25.243757 10887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41131 every 8 connection(s)
I20260927 07:09:25.245313 10755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:09:25.251844 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 10755
I20260927 07:09:25.252107 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:09:25.252948 30785 external_mini_cluster.cc:1428] Setting key 3306386f09a5a83c360236ea5ffff296
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:25.284394 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:25.250288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:34861 (local address 127.30.16.126:36417)
0927 07:09:25.259068 (+ 8780us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:25.259075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:25.260352 (+ 1277us) server_negotiation.cc:407] Connection header received
0927 07:09:25.263190 (+ 2838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:25.263199 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:25.263306 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:25.263392 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:25.265991 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:25.267270 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:25.270071 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:25.270847 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:25.270972 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:25.276049 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:25.276058 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:25.276066 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:25.276115 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:25.280019 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:25.281070 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:25.281078 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:25.281085 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:25.281246 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:25.281823 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:25.281829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:25.281832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:25.282295 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:25.282422 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:25.283657 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0927 07:09:25.283947 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8641,"thread_start_us":8481,"threads_started":1}
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:09:25.287407 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:25.249538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.65:34861)
0927 07:09:25.259659 (+ 10121us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:25.259716 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:25.262813 (+ 3097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:25.264016 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:25.264032 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:25.264797 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:25.265737 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:25.265769 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:25.267476 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:25.267484 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:25.269870 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:25.269888 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:25.271327 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:25.271336 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:25.271670 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:25.272567 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:25.272602 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:25.275739 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:25.280204 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:25.280217 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:25.280241 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:25.280924 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:25.281402 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:25.281406 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:25.281410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:25.281674 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:25.282541 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:25.282578 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:25.283135 (+ 557us) client_negotiation.cc:769] Sending connection context
0927 07:09:25.285023 (+ 1888us) client_negotiation.cc:240] Negotiation successful
0927 07:09:25.285636 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9765,"thread_start_us":10374,"threads_started":1}
I20260927 07:09:25.289305 10888 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:25.289675 10888 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:25.290782 10888 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:09:25.293479 10675 ts_manager.cc:194] Registered new tserver with Master: e40308d95d3247459639d50109706a9e (127.30.16.65:41131)
I20260927 07:09:25.295887 10675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:34861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:09:25.322862 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492964, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
W20260927 07:09:25.805289 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:25.805785 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:25.806001 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:25.817963 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:25.818217 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:25.818486 10894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:09:25.834646 10894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:25.836762 10894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:25.838506 10894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:25.869676 10901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:25.870806 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:25.877910 10894 server_base.cc:1034] running on GCE node
W20260927 07:09:25.878748 10905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:25.880585 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:25.886638 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:25.887979 10894 hybrid_clock.cc:648] HybridClock initialized: now 1790492965887872 us; error 110 us; skew 500 ppm
I20260927 07:09:25.888435 10894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:25.892835 10894 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:09:25.915052 10894 webserver.cc:460] Webserver started at http://127.30.16.66:36607/ using document root <none> and password file <none>
I20260927 07:09:25.916172 10894 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:25.916342 10894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:25.916739 10894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:09:25 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:25.948385 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "0a6299554fbc407f84bed2e38a53c8aa"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "fb25c5e4c5e83db2b2a6374b2867d2dc"
server_key_iv: "3689ddb718a2fe49a8e3772a8b0bc5c3"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.949648 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "0a6299554fbc407f84bed2e38a53c8aa"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "fb25c5e4c5e83db2b2a6374b2867d2dc"
server_key_iv: "3689ddb718a2fe49a8e3772a8b0bc5c3"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.969650 10894 fs_manager.cc:691] Time spent creating directory manager: real 0.020s user 0.008s sys 0.008s
I20260927 07:09:25.996052 10911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:25.997865 10894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260927 07:09:25.998095 10894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "0a6299554fbc407f84bed2e38a53c8aa"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "fb25c5e4c5e83db2b2a6374b2867d2dc"
server_key_iv: "3689ddb718a2fe49a8e3772a8b0bc5c3"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:25.998436 10894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:26.018836 10894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:26.024598 10894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:26.025027 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:26.026341 10894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:26.031590 10894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:26.031713 10894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:26.031875 10894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:26.031955 10894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:26.080516 10894 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:38957
I20260927 07:09:26.080551 11030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:38957 every 8 connection(s)
I20260927 07:09:26.083033 10894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:09:26.092733 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 10894
I20260927 07:09:26.092926 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:09:26.093492 30785 external_mini_cluster.cc:1428] Setting key d10fefceefc21798988c1d61024df8f6
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492965, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:26.109640 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.085824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:46301 (local address 127.30.16.126:36417)
0927 07:09:26.086328 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:26.086335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:26.087588 (+ 1253us) server_negotiation.cc:407] Connection header received
0927 07:09:26.090597 (+ 3009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:26.090604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:26.090703 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:26.090806 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:26.093144 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.094394 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.096510 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.097213 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.097356 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.102195 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:26.102203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:26.102209 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:26.102253 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:26.106305 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.107191 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.107195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.107197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.107319 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.107794 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.107798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.107800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.108086 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:26.108267 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:26.109016 (+ 749us) server_negotiation.cc:299] Negotiation successful
0927 07:09:26.109260 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
I20260927 07:09:26.110950 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.086411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.66:46301)
0927 07:09:26.087380 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:26.087433 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:26.090182 (+ 2749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:26.091154 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:26.091168 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:26.091925 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:26.092873 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:26.092903 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.094575 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.094582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:26.096338 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:26.096354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.097477 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.097486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:26.097833 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.098719 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:26.098755 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:26.101893 (+ 3138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:26.106482 (+ 4589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:26.106489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:26.106505 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:26.107014 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:26.107437 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:26.107443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:26.107447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:26.107660 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:26.108225 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:26.108234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:26.108615 (+ 381us) client_negotiation.cc:769] Sending connection context
0927 07:09:26.108897 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 07:09:26.109456 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":398,"threads_started":1}
I20260927 07:09:26.112449 11031 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:26.112785 11031 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:26.113519 11031 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:26.115049 10675 ts_manager.cc:194] Registered new tserver with Master: 0a6299554fbc407f84bed2e38a53c8aa (127.30.16.66:38957)
I20260927 07:09:26.115904 10675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:46301
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:09:26.159229 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:26.300071 10888 heartbeater.cc:503] Master 127.30.16.126:36417 was elected leader, sending a full tablet report...
W20260927 07:09:26.506114 11037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:26.506438 11037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:26.506629 11037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:26.514364 11037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:26.514484 11037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:26.514731 11037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:09:26.525208 11037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:26.526686 11037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:26.528147 11037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:26.546476 11045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:26.546566 11043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:26.546587 11042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:26.547472 11037 server_base.cc:1034] running on GCE node
I20260927 07:09:26.548202 11037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:26.549280 11037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:26.550536 11037 hybrid_clock.cc:648] HybridClock initialized: now 1790492966550524 us; error 119 us; skew 500 ppm
I20260927 07:09:26.550982 11037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:26.554207 11037 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:09:26.555792 11037 webserver.cc:460] Webserver started at http://127.30.16.67:36019/ using document root <none> and password file <none>
I20260927 07:09:26.556494 11037 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:26.556643 11037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:26.556962 11037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:26.570003 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "658036128a3f40a0a8d0eecd97ea2c31"
format_stamp: "Formatted at 2026-09-27 07:09:26 on dist-test-slave-8vqf"
server_key: "8db70aa41a48933cf927db3887c26aeb"
server_key_iv: "e80d27788962fa59952578bdf8aeac8a"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:26.570900 11037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "658036128a3f40a0a8d0eecd97ea2c31"
format_stamp: "Formatted at 2026-09-27 07:09:26 on dist-test-slave-8vqf"
server_key: "8db70aa41a48933cf927db3887c26aeb"
server_key_iv: "e80d27788962fa59952578bdf8aeac8a"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:26.576357 11037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:09:26.591372 11052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:26.592923 11037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:09:26.593096 11037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "658036128a3f40a0a8d0eecd97ea2c31"
format_stamp: "Formatted at 2026-09-27 07:09:26 on dist-test-slave-8vqf"
server_key: "8db70aa41a48933cf927db3887c26aeb"
server_key_iv: "e80d27788962fa59952578bdf8aeac8a"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:26.593313 11037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:26.609444 11037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:26.613759 11037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:26.614091 11037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:26.615128 11037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:26.616539 11037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:26.616609 11037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:26.616755 11037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:26.616803 11037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:26.649241 11037 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36547
I20260927 07:09:26.649281 11164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36547 every 8 connection(s)
I20260927 07:09:26.650671 11037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:09:26.656576 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11037
I20260927 07:09:26.656765 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:09:26.657418 30785 external_mini_cluster.cc:1428] Setting key a79d208e3062b916d30df112ade840c1
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492966, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:26.671597 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.653498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:43841 (local address 127.30.16.126:36417)
0927 07:09:26.653970 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:26.653974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:26.654944 (+ 970us) server_negotiation.cc:407] Connection header received
0927 07:09:26.656989 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:26.656994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:26.657085 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:26.657193 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:26.659109 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.660151 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.662109 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.662505 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.662707 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.666057 (+ 3350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:26.666062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:26.666066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:26.666098 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:26.668762 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.669406 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.669410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.669412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.669547 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.669906 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.669909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.669911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.670163 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:26.670278 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:26.670988 (+ 710us) server_negotiation.cc:299] Negotiation successful
0927 07:09:26.671254 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":235,"threads_started":1}
I20260927 07:09:26.672633 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.654064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.67:43841)
0927 07:09:26.654790 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:26.654833 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:26.656661 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:26.657482 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:26.657494 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:26.658066 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:26.658929 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:26.658958 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.660450 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.660454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:26.661957 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:26.661969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.662781 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.662785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:26.663041 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.663684 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:26.663707 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:26.665856 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:26.668876 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:26.668883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:26.668896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:26.669311 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:26.669642 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:26.669645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:26.669647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:26.669819 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:26.670275 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:26.670281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:26.670641 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 07:09:26.670877 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 07:09:26.671244 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":274,"threads_started":1}
I20260927 07:09:26.674062 11165 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:26.674383 11165 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:26.675155 11165 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:26.676515 10675 ts_manager.cc:194] Registered new tserver with Master: 658036128a3f40a0a8d0eecd97ea2c31 (127.30.16.67:36547)
I20260927 07:09:26.677305 10675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:43841
I20260927 07:09:26.685966 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:09:26.687798 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 10755
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:09:26.705394 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.688864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.30.16.65:41131)
0927 07:09:26.689472 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:26.689479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:26.689537 (+ 58us) server_negotiation.cc:407] Connection header received
0927 07:09:26.689612 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:26.689628 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:26.689905 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:26.690108 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:26.691705 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.692881 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.694353 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.695064 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.695161 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.699551 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:26.699563 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:26.699577 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:26.699616 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:26.702440 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.703203 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.703215 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.703219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.703386 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.703794 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.703799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.703803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.704148 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:26.704453 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:26.704732 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 07:09:26.705028 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":302,"threads_started":1}
I20260927 07:09:26.706701 10883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:26.796123 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 10755
I20260927 07:09:26.812716 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 10894
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:09:26.828656 11172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.813599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.30.16.66:38957)
0927 07:09:26.814150 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:26.814157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:26.814184 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:09:26.814251 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:26.814257 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:26.814467 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:26.814684 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:26.815892 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.816859 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.818323 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.818943 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.819040 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.823376 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:26.823388 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:26.823402 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:26.823438 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:26.826126 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.826843 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.826850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.826852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.826989 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.827324 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.827327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.827329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.827659 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:26.827865 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:26.828159 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 07:09:26.828362 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":285,"threads_started":1}
I20260927 07:09:26.830050 11026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:26.914247 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 10894
I20260927 07:09:26.930208 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11037
Sep 27 07:09:26 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492963, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:09:26.947464 11174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:26.931435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.30.16.67:36547)
0927 07:09:26.932083 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:26.932090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:26.932121 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:09:26.932180 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:26.932186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:26.932411 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:26.932568 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:26.933828 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.934930 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.936210 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:26.936913 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:26.937059 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:26.941680 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:26.941692 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:26.941709 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:26.941751 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:26.944597 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.945282 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.945289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.945291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.945421 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:26.945770 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:26.945774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:26.945777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:26.946188 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:26.946451 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:26.946910 (+ 459us) server_negotiation.cc:299] Negotiation successful
0927 07:09:26.947169 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":334,"threads_started":1}
I20260927 07:09:26.948835 11160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:27.034715 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11037
I20260927 07:09:27.051189 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 10643
I20260927 07:09:27.052529 10705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:27.159446 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 10643
I20260927 07:09:27.184558 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36417
--webserver_interface=127.30.16.126
--webserver_port=43867
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:36417
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:27.524976 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:27.525296 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:27.525388 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:27.534109 11177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:27.534246 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:27.534296 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:27.534358 11177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:27.534409 11177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:27.545703 11177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:36417
--ranger_config_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:36417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=43867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:27.547086 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:27.548393 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:27.560503 11182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:27.560564 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:27.560542 11183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:27.561354 11177 server_base.cc:1034] running on GCE node
I20260927 07:09:27.562191 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:27.563743 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:27.564944 11177 hybrid_clock.cc:648] HybridClock initialized: now 1790492967564883 us; error 70 us; skew 500 ppm
I20260927 07:09:27.565346 11177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:27 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:27.568620 11177 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:27.570087 11177 webserver.cc:460] Webserver started at http://127.30.16.126:43867/ using document root <none> and password file <none>
I20260927 07:09:27.570842 11177 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:27.570976 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:09:27 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492967, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:27.641237 11177 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 07:09:27.644415 11193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:27.645829 11177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:09:27.646025 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "9c2bdd8d10134fb0be068587ff34ad65"
format_stamp: "Formatted at 2026-09-27 07:09:23 on dist-test-slave-8vqf"
server_key: "777e8ea35cce11dbe73b03b3f6b601f4"
server_key_iv: "2a53bde4b9965b9fe2aaeb289ff02721"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:27.647123 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:27.675406 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:27.679759 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:27.680083 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:27.698819 11177 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:36417
I20260927 07:09:27.698837 11244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:36417 every 8 connection(s)
I20260927 07:09:27.700397 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:27.707489 11245 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:09:27.708755 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11177
I20260927 07:09:27.709830 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:41131
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:36417
--webserver_port=35049
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:27.712342 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Bootstrap starting.
I20260927 07:09:27.733306 11245 log.cc:824] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:27.742871 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:09:27.743428 11245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Bootstrap complete.
I20260927 07:09:27.752871 11245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:27.753733 11245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2bdd8d10134fb0be068587ff34ad65, State: Initialized, Role: FOLLOWER
I20260927 07:09:27.754510 11245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:27.754820 11245 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:27.754949 11245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:27.755146 11245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:09:27.759120 11245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:27.759616 11245 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2bdd8d10134fb0be068587ff34ad65; no voters:
I20260927 07:09:27.760267 11245 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:09:27.760510 11249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:09:27.761793 11249 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [term 2 LEADER]: Becoming Leader. State: Replica: 9c2bdd8d10134fb0be068587ff34ad65, State: Running, Role: LEADER
I20260927 07:09:27.762475 11245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:27.762662 11249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } }
I20260927 07:09:27.765666 11251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2bdd8d10134fb0be068587ff34ad65. Latest consensus state: current_term: 2 leader_uuid: "9c2bdd8d10134fb0be068587ff34ad65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } } }
I20260927 07:09:27.766057 11251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:27.766585 11250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9c2bdd8d10134fb0be068587ff34ad65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2bdd8d10134fb0be068587ff34ad65" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 36417 } } }
I20260927 07:09:27.766847 11250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:27.774287 11257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:27.780139 11257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:27.781591 11257 catalog_manager.cc:1259] Loaded cluster ID: 1eef9c7d664e47c38197c0093b103ffb
I20260927 07:09:27.781746 11257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:27.785607 11257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:27.787204 11257 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Loaded TSK: 0
I20260927 07:09:27.788872 11257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:09:27.792027 11245 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 07:09:28.143225 11247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:28.143559 11247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:28.143743 11247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:28.153600 11247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:28.153836 11247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:28.154075 11247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:09:28.170013 11247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:41131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=35049
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:28.172082 11247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:28.174000 11247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:28.242776 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:28.243886 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:28.245342 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:28.246594 11247 server_base.cc:1034] running on GCE node
I20260927 07:09:28.247294 11247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:28.248507 11247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:28.249765 11247 hybrid_clock.cc:648] HybridClock initialized: now 1790492968249686 us; error 63 us; skew 500 ppm
I20260927 07:09:28.250121 11247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:28 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:28.253667 11247 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:09:28.290830 11247 webserver.cc:460] Webserver started at http://127.30.16.65:35049/ using document root <none> and password file <none>
I20260927 07:09:28.292049 11247 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:28.292258 11247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:09:28 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:28.320694 11247 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:09:28.323978 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:28.336374 11247 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260927 07:09:28.336730 11247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e40308d95d3247459639d50109706a9e"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "192c1245238f82161c281cc075d5d8bc"
server_key_iv: "4fbcdc375426c54d8741ecac953284c7"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:28.338356 11247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:28.396509 11247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:28.401887 11247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:28.402288 11247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:28.407397 11247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:28.409237 11247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:28.409332 11247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:28.409493 11247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:28.409572 11247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:28.474920 11247 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:41131
I20260927 07:09:28.474987 11407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:41131 every 8 connection(s)
I20260927 07:09:28.476745 11247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:09:28.484442 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11247
I20260927 07:09:28.485242 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:38957
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:36417
--webserver_port=36607
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:28 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492968, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:28.517692 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:28.480371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.65:36257)
0927 07:09:28.481265 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:28.481318 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:28.483384 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:28.498298 (+ 14914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:28.498315 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:28.498990 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:28.499628 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:28.499649 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:28.501460 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:28.501467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:28.503036 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:28.503051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:28.504294 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:28.504302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:28.504763 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:28.505538 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:28.505569 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:28.508078 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:28.512073 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:28.512083 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:28.512102 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:28.512579 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:28.513096 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:28.513101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:28.513106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:28.513339 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:28.514599 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:28.514607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:28.515100 (+ 493us) client_negotiation.cc:769] Sending connection context
0927 07:09:28.515470 (+ 370us) client_negotiation.cc:240] Negotiation successful
0927 07:09:28.516112 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":381,"threads_started":1}
I20260927 07:09:28.517736 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:28.481340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:36257 (local address 127.30.16.126:36417)
0927 07:09:28.496115 (+ 14775us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:28.496146 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:28.496218 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:09:28.496857 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:28.496885 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:28.497551 (+ 666us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:28.498100 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:28.499871 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:28.501254 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:28.503212 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:28.504003 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:28.504237 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:28.508393 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:28.508401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:28.508418 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:28.508462 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:28.511892 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:28.512762 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:28.512769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:28.512776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:28.512947 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:28.513478 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:28.513485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:28.513492 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:28.514258 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:28.514481 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:28.515724 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0927 07:09:28.516121 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14483,"thread_start_us":14239,"threads_started":1}
I20260927 07:09:28.522023 11408 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:28.522369 11408 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:28.523201 11408 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
W20260927 07:09:28.526152 11408 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:36417 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:09:28.526396 11408 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:28.526750 11408 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:28.527982 11408 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:28.528254 11408 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
W20260927 07:09:28.529428 11408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:09:28.925143 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:28.925485 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:28.925666 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:28.933341 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:28.933504 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:28.933679 11412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:09:28.944934 11412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:38957
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=36607
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:28.946435 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:28.947851 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:29.001551 11428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:29.005095 11426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:29.011251 11412 server_base.cc:1034] running on GCE node
W20260927 07:09:29.011552 11431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:29.012468 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:29.015005 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:29.016317 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790492969016245 us; error 64 us; skew 500 ppm
I20260927 07:09:29.016713 11412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:29.025422 11412 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:09:29.027652 11412 webserver.cc:460] Webserver started at http://127.30.16.66:36607/ using document root <none> and password file <none>
I20260927 07:09:29.028714 11412 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:29.028892 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:29.055465 11412 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:09:29.058899 11439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:29.060518 11412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 07:09:29.060776 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "0a6299554fbc407f84bed2e38a53c8aa"
format_stamp: "Formatted at 2026-09-27 07:09:25 on dist-test-slave-8vqf"
server_key: "fb25c5e4c5e83db2b2a6374b2867d2dc"
server_key_iv: "3689ddb718a2fe49a8e3772a8b0bc5c3"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:29.061981 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:29.078679 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:29.083974 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:29.084388 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:29.091370 11412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:29.099472 11412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:29.099629 11412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:29.099761 11412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:29.099841 11412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:29.155046 11412 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:38957
I20260927 07:09:29.155133 11552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:38957 every 8 connection(s)
I20260927 07:09:29.156906 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:09:29.166029 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11412
I20260927 07:09:29.166816 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.30.16.84:59847/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:36547
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:36417
--webserver_port=36019
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:44101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:29.218436 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.161081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:59181 (local address 127.30.16.126:36417)
0927 07:09:29.196486 (+ 35405us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:29.196493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:29.200691 (+ 4198us) server_negotiation.cc:407] Connection header received
0927 07:09:29.200777 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:29.200783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:29.200880 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:29.201018 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:29.203986 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.205359 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.208140 (+ 2781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.208591 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.208752 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:29.212252 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:29.212257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:29.212260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:29.212293 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:29.215207 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.215978 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.215987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.215990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.216111 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.216602 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.216607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.216610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.216893 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:29.217052 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:29.217820 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 07:09:29.218042 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35252,"thread_start_us":35422,"threads_started":1}
I20260927 07:09:29.219590 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.160462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.66:59181)
0927 07:09:29.197286 (+ 36824us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:29.197362 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:29.200163 (+ 2801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:29.201487 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:29.201507 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:29.202455 (+ 948us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:29.203668 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:29.203703 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.205633 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.205642 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:29.207992 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:29.208009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.208864 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.208888 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:29.209162 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:29.209878 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:29.209906 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:29.212000 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:29.215355 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:29.215365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:29.215384 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:29.215829 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:29.216272 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:29.216275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:29.216277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:29.216480 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:29.217027 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:29.217035 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:29.217460 (+ 425us) client_negotiation.cc:769] Sending connection context
0927 07:09:29.217735 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 07:09:29.218310 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36423,"thread_start_us":45398,"threads_started":1}
I20260927 07:09:29.221264 11553 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:29.221608 11553 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:29.222491 11553 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:29.224570 11210 ts_manager.cc:194] Registered new tserver with Master: 0a6299554fbc407f84bed2e38a53c8aa (127.30.16.66:38957)
I20260927 07:09:29.226454 11210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:59181
I20260927 07:09:29.529583 11408 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:29.530025 11408 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:29.531473 11210 ts_manager.cc:194] Registered new tserver with Master: e40308d95d3247459639d50109706a9e (127.30.16.65:41131)
I20260927 07:09:29.532610 11210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:36257
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
W20260927 07:09:29.656095 11557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:29.656437 11557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:29.656617 11557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:29.664467 11557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:29.664616 11557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:29.664803 11557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:09:29.678841 11557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:44101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.30.16.84:59847/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:36547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=36019
--tserver_master_addrs=127.30.16.126:36417
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:29.680742 11557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:29.682626 11557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:29.698645 11563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:29.699388 11562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:29.699717 11565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:29.699757 11557 server_base.cc:1034] running on GCE node
I20260927 07:09:29.700409 11557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:29.701480 11557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:29.702699 11557 hybrid_clock.cc:648] HybridClock initialized: now 1790492969702623 us; error 46 us; skew 500 ppm
I20260927 07:09:29.703138 11557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:29.707733 11557 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:09:29.709707 11557 webserver.cc:460] Webserver started at http://127.30.16.67:36019/ using document root <none> and password file <none>
I20260927 07:09:29.710777 11557 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:29.710958 11557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:29.747956 11557 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.000s sys 0.010s
I20260927 07:09:29.752127 11573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:29.754007 11557 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260927 07:09:29.754200 11557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "658036128a3f40a0a8d0eecd97ea2c31"
format_stamp: "Formatted at 2026-09-27 07:09:26 on dist-test-slave-8vqf"
server_key: "8db70aa41a48933cf927db3887c26aeb"
server_key_iv: "e80d27788962fa59952578bdf8aeac8a"
server_key_version: "kuduclusterkey@0"
I20260927 07:09:29.755692 11557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:29.792167 11557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:29.797320 11557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:29.797648 11557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:29.798913 11557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:29.800911 11557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:29.801023 11557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:29.801133 11557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:29.801208 11557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for HTTP/127.30.16.84@KRBTEST.COM
I20260927 07:09:29.897203 11557 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36547
I20260927 07:09:29.897280 11686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36547 every 8 connection(s)
I20260927 07:09:29.899283 11557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:09:29.908521 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11557
Sep 27 07:09:29 dist-test-slave-8vqf krb5kdc[10199](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492969, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:29.944473 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.903370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:36417 (local address 127.30.16.67:44469)
0927 07:09:29.919548 (+ 16178us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:29.919619 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:29.921515 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:29.922532 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:29.922581 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:29.923455 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:29.924450 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:29.924477 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.926010 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.926017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:29.928164 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:29.928181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.929280 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.929288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:29.929589 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:29.931374 (+ 1785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:29.931410 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:29.934093 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:29.939523 (+ 5430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:29.939533 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:29.939554 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:29.940180 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:29.940622 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:29.940627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:29.940631 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:29.940902 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:29.941591 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:29.941600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:29.942066 (+ 466us) client_negotiation.cc:769] Sending connection context
0927 07:09:29.942371 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 07:09:29.942995 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15832,"thread_start_us":13465,"threads_started":1}
I20260927 07:09:29.945640 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.902539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:44469 (local address 127.30.16.126:36417)
0927 07:09:29.915235 (+ 12696us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:29.915245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:29.921862 (+ 6617us) server_negotiation.cc:407] Connection header received
0927 07:09:29.921931 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:29.921935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:29.922030 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:29.922161 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:29.924763 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.925783 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.928364 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.929012 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.929153 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:29.934385 (+ 5232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:29.934391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:29.934396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:29.934443 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:29.939348 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.940355 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.940360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.940364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.940492 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.941044 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.941049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.941052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.941387 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:29.941520 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:29.944992 (+ 3472us) server_negotiation.cc:299] Negotiation successful
0927 07:09:29.945252 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12483,"thread_start_us":12683,"threads_started":1}
I20260927 07:09:29.948019 11687 heartbeater.cc:347] Connected to a master server at 127.30.16.126:36417
I20260927 07:09:29.948346 11687 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:29.949371 11687 heartbeater.cc:511] Master 127.30.16.126:36417 requested a full tablet report, sending...
I20260927 07:09:29.951421 11210 ts_manager.cc:194] Registered new tserver with Master: 658036128a3f40a0a8d0eecd97ea2c31 (127.30.16.67:36547)
I20260927 07:09:29.952539 11210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:44469
I20260927 07:09:29.958765 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.915474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52444 (local address 127.30.16.126:36417)
0927 07:09:29.924368 (+ 8894us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:29.924375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:29.924396 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:09:29.924451 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:29.924456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:29.924554 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:29.924673 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:29.927191 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.928358 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.930348 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.930959 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:29.931090 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:29.935752 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:29.935759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:29.935763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:29.935811 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:29.939922 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.950675 (+ 10753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.950682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.950686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.950898 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:29.956845 (+ 5947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:29.956851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:29.956854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:29.957235 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:29.957527 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:29.957921 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 07:09:29.958319 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8733,"thread_start_us":9393,"threads_started":1}
I20260927 07:09:29.961700 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:09:30.012533 11692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:29.992281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.30.16.126:36417)
0927 07:09:29.992503 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:29.992513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:29.994415 (+ 1902us) server_negotiation.cc:407] Connection header received
0927 07:09:29.994604 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:29.994611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:29.994713 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:29.994842 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:29.996969 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:29.998140 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.000347 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.001003 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.001126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.004929 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:30.004937 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:30.004942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:30.004994 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:30.008885 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.009907 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.009914 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.009918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.010094 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.010673 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.010679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.010683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.011063 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:30.011306 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.011940 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.012193 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 07:09:30.020124 11210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:09:30.031553 11210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:09:30.111372 11704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.059674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:38957 (local address 127.0.0.1:42170)
0927 07:09:30.097133 (+ 37459us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.097187 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:30.097419 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:30.101400 (+ 3981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:30.101409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:30.101446 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:30.101890 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.101905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.104715 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.104721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.107145 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.107161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.110255 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.110262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.110486 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.110573 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 07:09:30.110732 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 07:09:30.110975 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37140,"mutex_wait_us":1,"thread_start_us":47591,"threads_started":1}
I20260927 07:09:30.112623 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.060231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42170 (local address 127.30.16.66:38957)
0927 07:09:30.100209 (+ 39978us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.100219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.100254 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:09:30.100383 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.100392 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.100692 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.100919 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:30.102874 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.104549 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.108267 (+ 3718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.109740 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.109868 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.110100 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.112131 (+ 2031us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.112307 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39799,"thread_start_us":40868,"threads_started":1}
I20260927 07:09:30.113921 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.070893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.30.16.67:36547)
0927 07:09:30.071706 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.071716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.102599 (+ 30883us) server_negotiation.cc:407] Connection header received
0927 07:09:30.102792 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.102801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.103051 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.103277 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:30.104174 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.105786 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.110878 (+ 5092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.112456 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.112553 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.112791 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.113385 (+ 594us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.113554 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":438,"threads_started":1}
I20260927 07:09:30.114846 11702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.070180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36547 (local address 127.0.0.1:40436)
0927 07:09:30.102443 (+ 32263us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.102464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:30.102611 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:30.103583 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:30.103588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:30.103628 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:30.103944 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.103955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.106753 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.106758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.110667 (+ 3909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.110684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.112991 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.112997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.113172 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.113207 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:09:30.114344 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0927 07:09:30.114490 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32117,"thread_start_us":27453,"threads_started":1}
I20260927 07:09:30.116884 11487 tablet_service.cc:1467] Processing CreateTablet for tablet 70004f17182e4e1990896af4124a9137 (DEFAULT_TABLE table=test-table [id=361ad69428724b01bf8231961a16ed05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:30.118346 11487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70004f17182e4e1990896af4124a9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:30.120211 11621 tablet_service.cc:1467] Processing CreateTablet for tablet 70004f17182e4e1990896af4124a9137 (DEFAULT_TABLE table=test-table [id=361ad69428724b01bf8231961a16ed05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:30.121834 11621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70004f17182e4e1990896af4124a9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:30.122834 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.076249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.30.16.65:41131)
0927 07:09:30.096591 (+ 20342us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.096604 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.108089 (+ 11485us) server_negotiation.cc:407] Connection header received
0927 07:09:30.108256 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.108264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.108558 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.108771 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:30.114516 (+ 5745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.116280 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.119801 (+ 3521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.121278 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.121410 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.121638 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.122273 (+ 635us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.122450 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20056,"thread_start_us":20355,"threads_started":1}
I20260927 07:09:30.123669 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.075913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36186)
0927 07:09:30.107948 (+ 32035us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.107967 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:30.108109 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:30.110628 (+ 2519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:30.110632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:30.110660 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:30.114277 (+ 3617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.114292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.117513 (+ 3221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.117520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.119592 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.119609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.121845 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.121854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.122056 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.122090 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:09:30.123119 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0927 07:09:30.123282 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31897,"mutex_wait_us":3224,"thread_start_us":33246,"threads_started":1}
I20260927 07:09:30.127674 11343 tablet_service.cc:1467] Processing CreateTablet for tablet 70004f17182e4e1990896af4124a9137 (DEFAULT_TABLE table=test-table [id=361ad69428724b01bf8231961a16ed05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:30.129243 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70004f17182e4e1990896af4124a9137. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:30.142465 11708 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Bootstrap starting.
I20260927 07:09:30.146067 11708 tablet_bootstrap.cc:654] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:30.147363 11708 log.cc:824] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:30.154361 11710 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Bootstrap starting.
I20260927 07:09:30.159266 11712 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Bootstrap starting.
I20260927 07:09:30.162420 11710 tablet_bootstrap.cc:654] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:30.166992 11708 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: No bootstrap required, opened a new log
I20260927 07:09:30.167333 11708 ts_tablet_manager.cc:1397] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.006s
I20260927 07:09:30.167534 11710 log.cc:824] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:30.168493 11712 tablet_bootstrap.cc:654] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:30.172163 11712 log.cc:824] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:30.178088 11708 raft_consensus.cc:348] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.178665 11708 raft_consensus.cc:374] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:30.178821 11708 raft_consensus.cc:729] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658036128a3f40a0a8d0eecd97ea2c31, State: Initialized, Role: FOLLOWER
I20260927 07:09:30.179582 11708 consensus_queue.cc:260] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.182096 11710 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: No bootstrap required, opened a new log
I20260927 07:09:30.182631 11710 ts_tablet_manager.cc:1397] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20260927 07:09:30.188221 11708 ts_tablet_manager.cc:1428] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Time spent starting tablet: real 0.021s user 0.013s sys 0.006s
I20260927 07:09:30.188735 11687 heartbeater.cc:503] Master 127.30.16.126:36417 was elected leader, sending a full tablet report...
I20260927 07:09:30.199630 11712 tablet_bootstrap.cc:492] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: No bootstrap required, opened a new log
I20260927 07:09:30.200064 11712 ts_tablet_manager.cc:1397] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Time spent bootstrapping tablet: real 0.041s user 0.014s sys 0.017s
I20260927 07:09:30.201445 11710 raft_consensus.cc:348] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.201934 11710 raft_consensus.cc:374] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:30.202072 11710 raft_consensus.cc:729] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40308d95d3247459639d50109706a9e, State: Initialized, Role: FOLLOWER
I20260927 07:09:30.202816 11710 consensus_queue.cc:260] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.203956 11408 heartbeater.cc:503] Master 127.30.16.126:36417 was elected leader, sending a full tablet report...
I20260927 07:09:30.204666 11710 ts_tablet_manager.cc:1428] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Time spent starting tablet: real 0.022s user 0.013s sys 0.003s
I20260927 07:09:30.211666 11712 raft_consensus.cc:348] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.212234 11712 raft_consensus.cc:374] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:30.212349 11712 raft_consensus.cc:729] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6299554fbc407f84bed2e38a53c8aa, State: Initialized, Role: FOLLOWER
I20260927 07:09:30.213037 11712 consensus_queue.cc:260] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.214124 11553 heartbeater.cc:503] Master 127.30.16.126:36417 was elected leader, sending a full tablet report...
I20260927 07:09:30.215001 11712 ts_tablet_manager.cc:1428] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
W20260927 07:09:30.229707 11409 tablet.cc:2369] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:09:30.247264 11723 raft_consensus.cc:482] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:30.247578 11723 raft_consensus.cc:504] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.249347 11723 leader_election.cc:288] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 658036128a3f40a0a8d0eecd97ea2c31 (127.30.16.67:36547), e40308d95d3247459639d50109706a9e (127.30.16.65:41131)
I20260927 07:09:30.257563 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.250193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:58073 (local address 127.30.16.65:41131)
0927 07:09:30.250392 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.250398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.251267 (+ 869us) server_negotiation.cc:407] Connection header received
0927 07:09:30.251520 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.251525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.251599 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.251680 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:30.252463 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.253543 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.255788 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.256546 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.256735 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.256810 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.257164 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.257301 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:09:30.257650 11724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.249836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.30.16.66:58073)
0927 07:09:30.251154 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.251171 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:30.251370 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:30.251802 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:30.251807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:30.251837 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:30.252279 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.252293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.253695 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.253700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.255647 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.255656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.256749 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.256755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.256957 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.257015 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 07:09:30.257126 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:09:30.257255 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"mutex_wait_us":1,"thread_start_us":614,"threads_started":1}
I20260927 07:09:30.257761 11705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.249874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:53069 (local address 127.30.16.67:36547)
0927 07:09:30.250055 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.250063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.251010 (+ 947us) server_negotiation.cc:407] Connection header received
0927 07:09:30.251357 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.251361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.251419 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.251531 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:30.252211 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.252973 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.255446 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.256635 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.256804 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.256877 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.257357 (+ 480us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.257498 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:09:30.258431 11725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.249563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36547 (local address 127.30.16.66:53069)
0927 07:09:30.250891 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.250912 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:30.251158 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:30.251569 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:30.251574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:30.251600 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:30.252044 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.252057 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.253108 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.253113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.255265 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:30.255280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.256966 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.256971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:30.257194 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.257223 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 07:09:30.257989 (+ 766us) client_negotiation.cc:240] Negotiation successful
0927 07:09:30.258125 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":47,"thread_start_us":1307,"threads_started":1}
I20260927 07:09:30.258960 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70004f17182e4e1990896af4124a9137" candidate_uuid: "0a6299554fbc407f84bed2e38a53c8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40308d95d3247459639d50109706a9e" is_pre_election: true
I20260927 07:09:30.259500 11363 raft_consensus.cc:2454] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6299554fbc407f84bed2e38a53c8aa in term 0.
I20260927 07:09:30.259466 11641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70004f17182e4e1990896af4124a9137" candidate_uuid: "0a6299554fbc407f84bed2e38a53c8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658036128a3f40a0a8d0eecd97ea2c31" is_pre_election: true
I20260927 07:09:30.259999 11641 raft_consensus.cc:2454] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a6299554fbc407f84bed2e38a53c8aa in term 0.
I20260927 07:09:30.260257 11443 leader_election.cc:302] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6299554fbc407f84bed2e38a53c8aa, e40308d95d3247459639d50109706a9e; no voters:
I20260927 07:09:30.260692 11723 raft_consensus.cc:2781] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:09:30.260865 11723 raft_consensus.cc:482] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:09:30.260964 11723 raft_consensus.cc:3037] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:30.264393 11723 raft_consensus.cc:504] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.265113 11723 leader_election.cc:288] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [CANDIDATE]: Term 1 election: Requested vote from peers 658036128a3f40a0a8d0eecd97ea2c31 (127.30.16.67:36547), e40308d95d3247459639d50109706a9e (127.30.16.65:41131)
I20260927 07:09:30.265600 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70004f17182e4e1990896af4124a9137" candidate_uuid: "0a6299554fbc407f84bed2e38a53c8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e40308d95d3247459639d50109706a9e"
I20260927 07:09:30.265594 11641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70004f17182e4e1990896af4124a9137" candidate_uuid: "0a6299554fbc407f84bed2e38a53c8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "658036128a3f40a0a8d0eecd97ea2c31"
I20260927 07:09:30.265807 11363 raft_consensus.cc:3037] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:30.265807 11641 raft_consensus.cc:3037] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:30.268362 11363 raft_consensus.cc:2454] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6299554fbc407f84bed2e38a53c8aa in term 1.
I20260927 07:09:30.268807 11443 leader_election.cc:302] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a6299554fbc407f84bed2e38a53c8aa, e40308d95d3247459639d50109706a9e; no voters:
I20260927 07:09:30.269062 11641 raft_consensus.cc:2454] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a6299554fbc407f84bed2e38a53c8aa in term 1.
I20260927 07:09:30.269119 11723 raft_consensus.cc:2781] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:30.269755 11723 raft_consensus.cc:686] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 1 LEADER]: Becoming Leader. State: Replica: 0a6299554fbc407f84bed2e38a53c8aa, State: Running, Role: LEADER
I20260927 07:09:30.270324 11723 consensus_queue.cc:237] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } }
I20260927 07:09:30.274478 11210 catalog_manager.cc:5495] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a6299554fbc407f84bed2e38a53c8aa (127.30.16.66). New cstate: current_term: 1 leader_uuid: "0a6299554fbc407f84bed2e38a53c8aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a6299554fbc407f84bed2e38a53c8aa" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 38957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 } health_report { overall_health: UNKNOWN } } }
I20260927 07:09:30.338212 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.331897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42178 (local address 127.30.16.66:38957)
0927 07:09:30.332119 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.332124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.332143 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:09:30.332340 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.332344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.332438 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.332575 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:09:30.333032 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.333815 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.335830 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.336269 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.336365 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.336978 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:09:30.337541 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:09:30.337711 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.337797 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.337868 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 07:09:30.347024 11363 raft_consensus.cc:1261] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 1 FOLLOWER]: Refusing update from remote peer 0a6299554fbc407f84bed2e38a53c8aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:30.347024 11641 raft_consensus.cc:1261] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 1 FOLLOWER]: Refusing update from remote peer 0a6299554fbc407f84bed2e38a53c8aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:30.348098 11726 consensus_queue.cc:1035] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "658036128a3f40a0a8d0eecd97ea2c31" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:30.348521 11723 consensus_queue.cc:1035] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e40308d95d3247459639d50109706a9e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 41131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:30.367434 11729 mvcc.cc:205] Tried to move back new op lower bound from 7333859206530740224 to 7333859206231302144. Current Snapshot: MvccSnapshot[applied={T|T < 7333859206530740224}]
I20260927 07:09:30.390152 11210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:30.390470 11210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:30.394861 11210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2bdd8d10134fb0be068587ff34ad65: Sending DeleteTablet for 3 replicas of tablet 70004f17182e4e1990896af4124a9137
I20260927 07:09:30.396142 11621 tablet_service.cc:1514] Processing DeleteTablet for tablet 70004f17182e4e1990896af4124a9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:40436
I20260927 07:09:30.396142 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet 70004f17182e4e1990896af4124a9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:36186
I20260927 07:09:30.396152 11487 tablet_service.cc:1514] Processing DeleteTablet for tablet 70004f17182e4e1990896af4124a9137 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:42170
I20260927 07:09:30.408206 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11247
I20260927 07:09:30.419277 11739 tablet_replica.cc:330] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: stopping tablet replica
I20260927 07:09:30.419974 11739 raft_consensus.cc:2229] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:09:30.420802 11739 raft_consensus.cc:2258] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:30.421406 11738 tablet_replica.cc:330] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: stopping tablet replica
I20260927 07:09:30.422222 11738 raft_consensus.cc:2229] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:30.424666 11739 ts_tablet_manager.cc:1905] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:30.425256 11738 raft_consensus.cc:2258] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:30.428375 11707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.409646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.30.16.65:41131)
0927 07:09:30.409963 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.409973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.409998 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:09:30.410078 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.410086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.410203 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.410322 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:30.412192 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.413216 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.415358 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.416273 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.416427 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.420465 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:30.420483 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:30.420502 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:30.420556 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:30.424394 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.425317 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.425327 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.425331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.425481 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.425899 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.425905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.425908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.426340 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:30.426647 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.427789 (+ 1142us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.428017 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260927 07:09:30.429380 11403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:30.433334 11739 ts_tablet_manager.cc:1918] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:30.435094 11738 ts_tablet_manager.cc:1905] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:30.442818 11738 ts_tablet_manager.cc:1918] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:30.443193 11739 log.cc:1196] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/70004f17182e4e1990896af4124a9137
I20260927 07:09:30.443903 11739 ts_tablet_manager.cc:1939] T 70004f17182e4e1990896af4124a9137 P 0a6299554fbc407f84bed2e38a53c8aa: Deleting consensus metadata
I20260927 07:09:30.444146 11738 log.cc:1196] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/70004f17182e4e1990896af4124a9137
I20260927 07:09:30.445904 11738 ts_tablet_manager.cc:1939] T 70004f17182e4e1990896af4124a9137 P 658036128a3f40a0a8d0eecd97ea2c31: Deleting consensus metadata
I20260927 07:09:30.447521 11194 catalog_manager.cc:4841] TS 0a6299554fbc407f84bed2e38a53c8aa (127.30.16.66:38957): tablet 70004f17182e4e1990896af4124a9137 (table test-table [id=361ad69428724b01bf8231961a16ed05]) successfully deleted
I20260927 07:09:30.448207 11195 catalog_manager.cc:4841] TS 658036128a3f40a0a8d0eecd97ea2c31 (127.30.16.67:36547): tablet 70004f17182e4e1990896af4124a9137 (table test-table [id=361ad69428724b01bf8231961a16ed05]) successfully deleted
I20260927 07:09:30.588467 11740 tablet_replica.cc:330] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e: stopping tablet replica
I20260927 07:09:30.588672 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11247
I20260927 07:09:30.588984 11740 raft_consensus.cc:2229] T 70004f17182e4e1990896af4124a9137 P e40308d95d3247459639d50109706a9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:30.606092 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11412
W20260927 07:09:30.606145 11197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.16.65:41131 (error 108)
I20260927 07:09:30.608435 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.607798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36212)
0927 07:09:30.608000 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.608124 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 07:09:30.608733 11197 catalog_manager.cc:4584] TS e40308d95d3247459639d50109706a9e (127.30.16.65:41131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 70004f17182e4e1990896af4124a9137: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
I20260927 07:09:30.621485 11703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.606931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.30.16.66:38957)
0927 07:09:30.607251 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.607258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.607544 (+ 286us) server_negotiation.cc:407] Connection header received
0927 07:09:30.607829 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.607836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.607966 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.608098 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:30.609593 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.610420 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.611824 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.612274 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.612417 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.615172 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:30.615186 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:30.615241 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:30.615287 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:30.618463 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.619539 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.619547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.619549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.619689 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.620079 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.620082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.620084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.620422 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:30.620704 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.620997 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.621163 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:09:30.622197 11548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:30.662540 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.662005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36222)
0927 07:09:30.662224 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.662350 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 07:09:30.663687 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.663293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36238)
0927 07:09:30.663444 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.663552 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 07:09:30.724260 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.723666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36244)
0927 07:09:30.723936 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.724050 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 07:09:30.725402 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.725028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36246)
0927 07:09:30.725194 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.725276 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 07:09:30.741896 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11412
I20260927 07:09:30.759882 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11557
I20260927 07:09:30.774132 11744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.760647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.30.16.67:36547)
0927 07:09:30.761217 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:30.761222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:30.761239 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:09:30.761292 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:30.761295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:30.761362 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:30.761515 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:30.763237 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.764003 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.765420 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:30.765796 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:30.765887 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:30.768626 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:30.768641 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:30.768657 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:30.768688 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:30.771560 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.772266 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.772273 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.772275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.772401 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:30.772738 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:30.772741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:30.772743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:30.773069 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:30.773294 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:30.773629 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 07:09:30.773790 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":233,"threads_started":1}
I20260927 07:09:30.775053 11682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:30.806273 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.805640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36256)
0927 07:09:30.805980 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.806093 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260927 07:09:30.807564 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.807035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36260)
0927 07:09:30.807279 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.807387 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260927 07:09:30.864434 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11557
I20260927 07:09:30.880676 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11177
I20260927 07:09:30.881916 11240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:30.996755 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11177
I20260927 07:09:30.996869 11706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:30.996326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:41131 (local address 127.0.0.1:36262)
0927 07:09:30.996556 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:30.996663 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:41131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 07:09:31.022096 30785 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 07:09:31.357894 30785 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 07:09:31.358104 30785 mini_ranger.cc:66] Stopping Ranger...
I20260927 07:09:31.704285 30785 mini_ranger.cc:68] Stopped Ranger
2026-09-27 07:09:31.704 UTC [10229] LOG: received smart shutdown request
2026-09-27 07:09:31.708 UTC [10229] LOG: background worker "logical replication launcher" (PID 10238) exited with exit code 1
2026-09-27 07:09:31.708 UTC [10233] LOG: shutting down
2026-09-27 07:09:31.893 UTC [10229] LOG: database system is shut down
2026-09-27T07:09:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53246 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:09:32 dist-test-slave-8vqf krb5kdc[11755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:09:32 dist-test-slave-8vqf krb5kdc[11755](info): set up 2 sockets
Sep 27 07:09:32 dist-test-slave-8vqf krb5kdc[11755](info): commencing operation
krb5kdc: starting...
W20260927 07:09:33.191031 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:09:33 dist-test-slave-8vqf krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:09:33.430346 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33363
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:33.767697 11768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:33.767999 11768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:33.768090 11768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:33.775300 11768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:33.775405 11768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:33.775452 11768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:33.775512 11768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:33.775555 11768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:33.785492 11768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:33.786846 11768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:33.788133 11768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:33.800036 11776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:33.800114 11774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:33.800093 11773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:33.800832 11768 server_base.cc:1034] running on GCE node
I20260927 07:09:33.801638 11768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:33.803212 11768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:33.804452 11768 hybrid_clock.cc:648] HybridClock initialized: now 1790492973804386 us; error 60 us; skew 500 ppm
I20260927 07:09:33.804821 11768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:33 dist-test-slave-8vqf krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:33.808125 11768 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:33.809572 11768 webserver.cc:460] Webserver started at http://127.30.16.126:38637/ using document root <none> and password file <none>
I20260927 07:09:33.810292 11768 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:33.810441 11768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:33.810797 11768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:33.813812 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "21160ff7c52341b89179695e8820ac14"
format_stamp: "Formatted at 2026-09-27 07:09:33 on dist-test-slave-8vqf"
server_key: "0756301d5918babf6e5150229c21cd49"
server_key_iv: "b5f150983cd55823f86b5c7e58514b45"
server_key_version: "encryptionkey@0"
I20260927 07:09:33.814744 11768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "21160ff7c52341b89179695e8820ac14"
format_stamp: "Formatted at 2026-09-27 07:09:33 on dist-test-slave-8vqf"
server_key: "0756301d5918babf6e5150229c21cd49"
server_key_iv: "b5f150983cd55823f86b5c7e58514b45"
server_key_version: "encryptionkey@0"
I20260927 07:09:33.820086 11768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 07:09:33.823793 11783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:33.825285 11768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:33.825493 11768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "21160ff7c52341b89179695e8820ac14"
format_stamp: "Formatted at 2026-09-27 07:09:33 on dist-test-slave-8vqf"
server_key: "0756301d5918babf6e5150229c21cd49"
server_key_iv: "b5f150983cd55823f86b5c7e58514b45"
server_key_version: "encryptionkey@0"
I20260927 07:09:33.825709 11768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:33.837124 11768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:33.841279 11768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:33.841600 11768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:33.861435 11768 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33363
I20260927 07:09:33.861477 11834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33363 every 8 connection(s)
I20260927 07:09:33.862985 11768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:33.863490 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11768
I20260927 07:09:33.863737 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:09:33.864655 30785 external_mini_cluster.cc:1428] Setting key 2d7c1a3773329095447b7a08b60be763
I20260927 07:09:33.868211 11835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:33.876477 11835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.30.16.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.125@KRBTEST.COM" created.
Sep 27 07:09:33 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:33.900492 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.879794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.30.16.126:33363)
0927 07:09:33.880681 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:33.880701 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:33.881129 (+ 428us) server_negotiation.cc:407] Connection header received
0927 07:09:33.883361 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:33.883366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:33.883624 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:33.883835 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:33.885237 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:33.886531 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:33.888174 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:33.888783 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:33.889002 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:33.893133 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:33.893143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:33.893156 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:33.893184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:33.895931 (+ 2747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:33.896584 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:33.896593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:33.896596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:33.896744 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:33.897134 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:33.897141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:33.897145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:33.897649 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:33.897863 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:33.898378 (+ 515us) server_negotiation.cc:299] Negotiation successful
0927 07:09:33.898827 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":626,"thread_start_us":408,"threads_started":1}
I20260927 07:09:33.900513 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.879798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33363 (local address 127.0.0.1:52784)
0927 07:09:33.880957 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:33.881005 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:33.882988 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:33.883862 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:33.883869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:33.884694 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:33.885042 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:33.885059 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:33.886738 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:33.886749 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:33.888016 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:33.888034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:33.889117 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:33.889125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:33.889330 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:33.890028 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:33.890065 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:33.892883 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:33.896065 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:33.896077 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:33.896090 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:33.896458 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:33.896851 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:33.896858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:33.896864 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:33.897032 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:33.897779 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:33.897793 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:33.898016 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 07:09:33.898264 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 07:09:33.898976 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":195,"spinlock_wait_cycles":19200,"thread_start_us":737,"threads_started":1}
I20260927 07:09:33.903966 11835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:09:33.905182 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.904651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59680)
0927 07:09:33.904861 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:33.905005 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 07:09:33.905450 11786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
I20260927 07:09:33.907330 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.906880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59696)
0927 07:09:33.907069 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:33.907161 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 07:09:33.907552 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:44847: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
I20260927 07:09:33.919284 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:44847
--webserver_interface=127.30.16.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:33.956929 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } attempt: 1
I20260927 07:09:33.958256 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.957711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59712)
0927 07:09:33.957942 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:33.958094 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 07:09:33.959349 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:33.958955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59720)
0927 07:09:33.959112 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:33.959193 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 07:09:33.959589 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:44847: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
I20260927 07:09:34.027879 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } attempt: 2
I20260927 07:09:34.029392 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.028744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59734)
0927 07:09:34.029079 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.029199 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 07:09:34.030761 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.030170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59736)
0927 07:09:34.030370 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.030469 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 07:09:34.031041 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:44847: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
I20260927 07:09:34.122359 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } attempt: 3
I20260927 07:09:34.123657 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.123163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59746)
0927 07:09:34.123385 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.123490 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 07:09:34.124838 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.124442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59756)
0927 07:09:34.124606 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.124696 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 07:09:34.125170 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:44847: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
W20260927 07:09:34.254383 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:34.254702 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:34.254941 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:34.262420 11842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:34.262526 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:34.262703 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:34.262801 11842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:34.262881 11842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:34.268450 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } attempt: 4
I20260927 07:09:34.269740 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.269195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59770)
0927 07:09:34.269499 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.269613 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260927 07:09:34.270911 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.270440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59774)
0927 07:09:34.270653 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.270766 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 07:09:34.271152 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:44847: Network error: Client connection negotiation failed: client connection to 127.30.16.125:44847: connect: Connection refused (error 111)
I20260927 07:09:34.273681 11842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:44847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.16.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:34.275144 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:34.276430 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:34.288532 11848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:34.288499 11847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:34.288725 11850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:34.289317 11842 server_base.cc:1034] running on GCE node
I20260927 07:09:34.290107 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:34.291738 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:34.293046 11842 hybrid_clock.cc:648] HybridClock initialized: now 1790492974292946 us; error 62 us; skew 500 ppm
I20260927 07:09:34.293416 11842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:34.303682 11842 init.cc:377] Logged in from keytab as kudu/127.30.16.125@KRBTEST.COM (short username kudu)
I20260927 07:09:34.305369 11842 webserver.cc:460] Webserver started at http://127.30.16.125:38191/ using document root <none> and password file <none>
I20260927 07:09:34.306123 11842 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:34.306272 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:34.306671 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:34.309900 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data/instance:
uuid: "a8d6973619c244d183ffed8de5a86c82"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "183c0e182e8271eb16ea4045846ce411"
server_key_iv: "dbd4286ec9dda4b61f84961bc6e87a1f"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.310788 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal/instance:
uuid: "a8d6973619c244d183ffed8de5a86c82"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "183c0e182e8271eb16ea4045846ce411"
server_key_iv: "dbd4286ec9dda4b61f84961bc6e87a1f"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.316365 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:09:34.320152 11857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:34.321712 11842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:09:34.321952 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "a8d6973619c244d183ffed8de5a86c82"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "183c0e182e8271eb16ea4045846ce411"
server_key_iv: "dbd4286ec9dda4b61f84961bc6e87a1f"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.322186 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:34.338773 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:34.343194 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:34.343540 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:34.362428 11842 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.125:44847
I20260927 07:09:34.362541 11908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.125:44847 every 8 connection(s)
I20260927 07:09:34.363883 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
I20260927 07:09:34.367564 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:34.372804 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11842
I20260927 07:09:34.373036 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-1/wal/instance
I20260927 07:09:34.373507 30785 external_mini_cluster.cc:1428] Setting key 3216243204a85bc13cc06a6fae46ce3b
I20260927 07:09:34.374200 11909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.124@KRBTEST.COM; defaulting to no policy
I20260927 07:09:34.396258 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.376815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52800 (local address 127.30.16.126:33363)
0927 07:09:34.377036 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.377042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.378320 (+ 1278us) server_negotiation.cc:407] Connection header received
0927 07:09:34.380910 (+ 2590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.380916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.381021 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.381166 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.382642 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.383801 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.385397 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.386191 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.386332 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.390271 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.390276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.390279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.390312 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.393102 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.393906 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.393913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.393916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.394079 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.394490 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.394498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.394502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.394856 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.395040 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.395834 (+ 794us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.396013 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.124@KRBTEST.COM" created.
I20260927 07:09:34.397598 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.377313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33363 (local address 127.0.0.1:52800)
0927 07:09:34.378129 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.378183 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.380637 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.381354 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.381361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.381877 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.382418 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.382435 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.383978 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.383982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.385240 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.385257 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.386430 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.386436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.386735 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.387504 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.387542 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.390035 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.393278 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.393287 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.393307 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.393783 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.394188 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.394193 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.394197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.394377 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.395021 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.395032 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.395463 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.395716 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.396229 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":305,"threads_started":1}
I20260927 07:09:34.399492 11909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:09:34.417881 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.400192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59780)
0927 07:09:34.400439 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.400461 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.400579 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.401907 (+ 1328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.401910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.402345 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.402479 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.402487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.404619 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.404627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.406090 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.406102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.407383 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.407387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.407563 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.408177 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.408194 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.410629 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.415132 (+ 4503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.415135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.415138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.415489 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.416077 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.416085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.416091 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.416252 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.417130 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.417134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.417258 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.417386 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.417615 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 07:09:34.418322 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.400526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.30.16.125:44847)
0927 07:09:34.401324 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.401335 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.401368 (+ 33us) server_negotiation.cc:407] Connection header received
0927 07:09:34.401428 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.401439 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.401726 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.401884 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.402706 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.404438 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.406229 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.407232 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.407375 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.410861 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.410880 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.410904 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.410949 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.414993 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.415692 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.415704 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.415710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.415871 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.416368 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.416376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.416382 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.416951 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.417191 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.417586 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.417893 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":586,"thread_start_us":433,"threads_started":1}
I20260927 07:09:34.420737 11909 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:09:34.421916 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.421486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42512)
0927 07:09:34.421639 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.421752 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 07:09:34.422216 11858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.423741 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.423438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42518)
0927 07:09:34.423566 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.423643 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 07:09:34.423930 11909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
I20260927 07:09:34.430949 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:35371
--webserver_interface=127.30.16.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:45755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:34.473263 11909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 1
I20260927 07:09:34.474643 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.474104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42530)
0927 07:09:34.474305 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.474420 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 07:09:34.475723 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.475371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42542)
0927 07:09:34.475524 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.475607 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 07:09:34.475980 11909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.544219 11909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 2
I20260927 07:09:34.550752 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.550152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42546)
0927 07:09:34.550335 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.550454 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 07:09:34.551875 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.551455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42558)
0927 07:09:34.551646 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.551733 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 07:09:34.552193 11909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.570427 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } attempt: 5
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:09:34.585641 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.571288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59782)
0927 07:09:34.571525 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.571543 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.571657 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.572009 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.572013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.572407 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.572605 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.572615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.574027 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.574032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.574989 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.575001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.576091 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.576096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.576250 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.576935 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.576955 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.579372 (+ 2417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.583264 (+ 3892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.583268 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.583271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.583647 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.584077 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.584081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.584083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.584210 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.584926 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.584930 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.585068 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.585185 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.585384 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 07:09:34.586151 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.571391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.30.16.125:44847)
0927 07:09:34.571673 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.571680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.571703 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:09:34.571770 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.571775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.571856 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.571982 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.572786 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.573867 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.575134 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.575906 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.576042 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.579575 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.579583 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.579588 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.579628 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.583131 (+ 3503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.583764 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.583770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.583773 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.583947 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.584328 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.584336 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.584341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.584794 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.584979 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.585358 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.585629 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 07:09:34.587283 11835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:09:34.588341 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.587921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42574)
0927 07:09:34.588144 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.588217 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 07:09:34.589262 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.588941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42590)
0927 07:09:34.589097 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.589172 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:09:34.589465 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.640698 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 1
I20260927 07:09:34.642022 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.641487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42602)
0927 07:09:34.641701 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.641798 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 07:09:34.643090 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.642737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42604)
0927 07:09:34.642886 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.642993 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:09:34.643294 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.643405 11909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 3
I20260927 07:09:34.644337 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.643910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42618)
0927 07:09:34.644119 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.644207 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 07:09:34.645565 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.645133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42624)
0927 07:09:34.645369 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.645455 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260927 07:09:34.645766 11909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.711544 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 2
I20260927 07:09:34.712762 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.712260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42634)
0927 07:09:34.712500 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.712594 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 07:09:34.713810 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.713460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42646)
0927 07:09:34.713639 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.713714 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 07:09:34.714025 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
W20260927 07:09:34.771988 11916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:34.772357 11916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:34.772523 11916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:34.779824 11916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:34.779974 11916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:34.780136 11916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:34.780241 11916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:34.780325 11916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:34.788959 11909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 4
I20260927 07:09:34.790242 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.789748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42654)
0927 07:09:34.789994 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.790084 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 07:09:34.790661 11916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:45755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33363,127.30.16.125:44847,127.30.16.124:35371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:35371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.16.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:34.791318 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.790991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42662)
0927 07:09:34.791150 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.791222 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 07:09:34.791518 11909 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.792212 11916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:34.793552 11916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:34.805747 11924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:34.805814 11922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:34.805840 11921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:34.806516 11916 server_base.cc:1034] running on GCE node
I20260927 07:09:34.807334 11916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:34.808758 11916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:34.809966 11916 hybrid_clock.cc:648] HybridClock initialized: now 1790492974809907 us; error 49 us; skew 500 ppm
I20260927 07:09:34.810340 11916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:34.813504 11916 init.cc:377] Logged in from keytab as kudu/127.30.16.124@KRBTEST.COM (short username kudu)
I20260927 07:09:34.815212 11916 webserver.cc:460] Webserver started at http://127.30.16.124:39551/ using document root <none> and password file <none>
I20260927 07:09:34.815964 11916 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:34.816116 11916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:34.816430 11916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:34.819695 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data/instance:
uuid: "dce28f14c8994ae7bd4410e42a07e992"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "74f59d5ee97608347b1e568a45e6321e"
server_key_iv: "9eb49ad1f0d1aa4d00eb461d3d530d95"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.820218 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 3
I20260927 07:09:34.820549 11916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal/instance:
uuid: "dce28f14c8994ae7bd4410e42a07e992"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "74f59d5ee97608347b1e568a45e6321e"
server_key_iv: "9eb49ad1f0d1aa4d00eb461d3d530d95"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.821282 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.820817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42674)
0927 07:09:34.821038 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.821134 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 07:09:34.822258 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.821897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42684)
0927 07:09:34.822101 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.822166 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 07:09:34.822438 11835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35371: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35371: connect: Connection refused (error 111)
I20260927 07:09:34.825933 11916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 07:09:34.829635 11931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:34.831168 11916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:09:34.831351 11916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "dce28f14c8994ae7bd4410e42a07e992"
format_stamp: "Formatted at 2026-09-27 07:09:34 on dist-test-slave-8vqf"
server_key: "74f59d5ee97608347b1e568a45e6321e"
server_key_iv: "9eb49ad1f0d1aa4d00eb461d3d530d95"
server_key_version: "encryptionkey@0"
I20260927 07:09:34.831574 11916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:34.853902 11916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:34.858088 11916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:34.858390 11916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:34.879354 11916 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.124:35371
I20260927 07:09:34.879377 11982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.124:35371 every 8 connection(s)
I20260927 07:09:34.880782 11916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
I20260927 07:09:34.884756 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:34.886812 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 11916
I20260927 07:09:34.887013 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790492767955610-30785-0/minicluster-data/master-2/wal/instance
I20260927 07:09:34.887588 30785 external_mini_cluster.cc:1428] Setting key 5edfb774c35c221e51347ca06fcc1834
I20260927 07:09:34.892906 11983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:34.911091 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.895949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.30.16.126:33363)
0927 07:09:34.896158 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.896164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.896960 (+ 796us) server_negotiation.cc:407] Connection header received
0927 07:09:34.897154 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.897160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.897235 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.897338 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.898411 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.899162 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.901186 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.901614 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.901712 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.905306 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.905314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.905319 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.905362 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.908427 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.909038 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.909045 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.909048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.909196 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.909598 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.909604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.909608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.909915 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.910075 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.910491 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.910745 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 07:09:34.911484 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.889570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.30.16.124:35371)
0927 07:09:34.890444 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.890455 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.890502 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:09:34.892210 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.892232 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.892624 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.892942 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.894739 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.896225 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.897861 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.898532 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.898802 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.903363 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.903373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.903388 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.903426 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.907138 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.907809 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.907820 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.907830 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.908010 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.908385 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.908389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.908392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.908946 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.909273 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.909686 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.910128 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":461,"threads_started":1}
I20260927 07:09:34.911513 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.895960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33363 (local address 127.0.0.1:52806)
0927 07:09:34.896805 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.896851 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.896997 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.897388 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.897392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.897962 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.898262 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.898270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.899337 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.899341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.901021 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.901036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.901859 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.901865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.901985 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.902621 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.902660 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.905079 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.908547 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.908556 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.908561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.908925 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.909322 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.909333 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.909337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.909487 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.910057 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.910063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.910216 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.910356 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.910641 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":446,"threads_started":1}
I20260927 07:09:34.913180 11983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:34.927121 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.913826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:44847 (local address 127.0.0.1:59796)
0927 07:09:34.914053 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.914070 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.914176 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.914520 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.914525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.914921 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.915054 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.915062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.916029 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.916037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.916895 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.916906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.917838 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.917843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.918008 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.918641 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.918659 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.921200 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.924548 (+ 3348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.924553 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.924557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.924956 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.925358 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.925363 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.925367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.925512 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.926055 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.926060 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.926249 (+ 189us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.926393 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.926717 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260927 07:09:34.927146 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.913953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.30.16.125:44847)
0927 07:09:34.914139 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.914145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.914166 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:09:34.914306 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.914312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.914396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.914494 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.915222 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.915894 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.917055 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.917695 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.917829 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.921390 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.921397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.921403 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.921465 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.924402 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.925093 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.925099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.925102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.925241 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.925642 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.925647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.925649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.925929 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.926139 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.926479 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.926749 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 07:09:34.928149 11983 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:09:34.931471 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.914632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52814 (local address 127.30.16.126:33363)
0927 07:09:34.914812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.914817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.914833 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:09:34.914994 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.914999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.915061 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.915166 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.916353 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.917109 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.918791 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.919483 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.919611 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.924656 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.924661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.924664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.924699 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.928296 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.929241 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.929244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.929247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.929423 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.929931 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.929936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.929939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.930328 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.930630 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.930913 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.931151 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
Sep 27 07:09:34 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:09:34.943938 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.928931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.30.16.124:35371)
0927 07:09:34.929100 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.929104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.929321 (+ 217us) server_negotiation.cc:407] Connection header received
0927 07:09:34.929430 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.929435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.929515 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.929596 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.930363 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.931430 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.933010 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.933687 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.933824 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.937662 (+ 3838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.937669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.937674 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.937710 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.941068 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.941720 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.941727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.941732 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.941875 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.942220 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.942226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.942230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.942678 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.942822 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.943255 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.943642 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":52}
I20260927 07:09:34.944165 11987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.928970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42706)
0927 07:09:34.929234 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:34.929259 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:34.929340 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:34.929646 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:34.929653 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:34.930095 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:34.930234 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.930242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.931580 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.931588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.932816 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:34.932827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.933825 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.933831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:34.934020 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.934717 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:34.934734 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:34.937435 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:34.941203 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.941207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.941209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.941575 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.941988 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:34.941991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:34.941993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:34.942114 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:34.942809 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:34.942812 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:34.942974 (+ 162us) client_negotiation.cc:769] Sending connection context
0927 07:09:34.943131 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 07:09:34.943494 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260927 07:09:34.950181 11983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: Bootstrap starting.
I20260927 07:09:34.951717 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.933243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.30.16.125:44847)
0927 07:09:34.933390 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.933395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.933458 (+ 63us) server_negotiation.cc:407] Connection header received
0927 07:09:34.933632 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.933635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.933690 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.933796 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.935479 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.936787 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.938736 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.939583 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.939720 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.945213 (+ 5493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.945222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.945229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.945275 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.948175 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.949150 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.949157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.949160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.949357 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.949866 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.949873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.949879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.950279 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.950632 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.951237 (+ 605us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.951424 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:09:34.954946 11983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:34.956477 11983 log.cc:824] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:34.957542 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:09:34.959597 11983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: No bootstrap required, opened a new log
I20260927 07:09:34.970381 11983 raft_consensus.cc:348] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:34.971082 11983 raft_consensus.cc:374] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:34.971268 11983 raft_consensus.cc:729] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce28f14c8994ae7bd4410e42a07e992, State: Initialized, Role: FOLLOWER
I20260927 07:09:34.972254 11983 consensus_queue.cc:260] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:34.978287 11996 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:34.978811 11996 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:34.980116 11983 sys_catalog.cc:564] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:34.984961 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.962542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.30.16.126:33363)
0927 07:09:34.962787 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.962795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.963922 (+ 1127us) server_negotiation.cc:407] Connection header received
0927 07:09:34.964222 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.964230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.964328 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.964420 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.967535 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.968951 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.971717 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.972688 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.972813 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.977034 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.977043 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.977048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.977109 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.980909 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.981771 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.981777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.981780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.981952 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.982449 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.982455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.982461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.982923 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.983213 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.984282 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.984546 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 07:09:34.987145 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.962948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.30.16.124:35371)
0927 07:09:34.963236 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.963244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.964164 (+ 920us) server_negotiation.cc:407] Connection header received
0927 07:09:34.964432 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.964438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.964530 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.964609 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.967074 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.968270 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.970680 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.971348 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.971452 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.976668 (+ 5216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.976678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.976686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.976744 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.981961 (+ 5217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.983271 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.983279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.983285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.983484 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.984083 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.984089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.984095 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.984595 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.984927 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.986268 (+ 1341us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.986633 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 07:09:34.992386 11913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:34.964548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.30.16.125:44847)
0927 07:09:34.964723 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:34.964730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:34.975203 (+ 10473us) server_negotiation.cc:407] Connection header received
0927 07:09:34.975806 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:34.975814 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:34.975908 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:34.975982 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:34.977806 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.979007 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.981013 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:34.981694 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:34.981788 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:34.985729 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:34.985735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:34.985740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:34.985787 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:34.989516 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.990228 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.990238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.990240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.990368 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:34.990818 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:34.990823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:34.990827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:34.991127 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:34.991392 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:34.991747 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 07:09:34.991978 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 07:09:34.996922 12007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:09:34.997114 12007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:09:34.999661 11835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 4
I20260927 07:09:34.994009 11916 master_runner.cc:186] Error getting master registration for 127.30.16.126:33363: OK, instance_id { permanent_uuid: "21160ff7c52341b89179695e8820ac14" instance_seqno: 1790492973855758 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:09:35.005872 11916 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 07:09:35 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492973, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:09:35.016620 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:35.000619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.30.16.124:35371)
0927 07:09:35.000793 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:35.000805 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:35.000822 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:09:35.000877 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:35.000882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:35.000954 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:35.001063 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:35.002041 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.003473 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.005397 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.006132 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.006277 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:35.010357 (+ 4080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:35.010366 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:35.010372 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:35.010417 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:35.013755 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:35.014457 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:35.014464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:35.014472 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:35.014678 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:35.015088 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:35.015093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:35.015097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:35.015391 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:35.015544 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:35.015990 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 07:09:35.016220 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 07:09:35.016642 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:35.000250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42726)
0927 07:09:35.000469 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:35.000492 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:35.000605 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:35.001137 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:35.001142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:35.001651 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:35.001873 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:35.001887 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.003681 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.003687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:35.005227 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:35.005246 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.006315 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.006321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:35.006514 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:35.007225 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:35.007248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:35.010045 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:35.013910 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:35.013917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:35.013922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:35.014322 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:35.014800 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:35.014805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:35.014809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:35.014963 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:35.015521 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:35.015526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:35.015722 (+ 196us) client_negotiation.cc:769] Sending connection context
0927 07:09:35.015901 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 07:09:35.016203 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 07:09:35.022114 11835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: Bootstrap starting.
I20260927 07:09:35.025563 11835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:35.026796 11835 log.cc:824] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:35.030356 11835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: No bootstrap required, opened a new log
I20260927 07:09:35.038007 11835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.038327 11835 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:35.038471 11835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21160ff7c52341b89179695e8820ac14, State: Initialized, Role: FOLLOWER
I20260927 07:09:35.039151 11835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.040135 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.040634 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.041592 11835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 07:09:35.052042 12021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:09:35.053433 12021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:09:35.051146 11768 master_runner.cc:186] Error getting master registration for 127.30.16.125:44847: OK, instance_id { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" instance_seqno: 1790492974357052 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:09:35.062840 11768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:09:35.090736 11909 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } attempt: 5
W20260927 07:09:35.095039 11789 tablet.cc:2369] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 07:09:35 dist-test-slave-8vqf krb5kdc[11755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492974, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:09:35.110321 11911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:35.091480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35371 (local address 127.0.0.1:42732)
0927 07:09:35.091771 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:35.091792 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:35.091915 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:35.092324 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:35.092329 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:35.092904 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:35.093132 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:35.093144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.096681 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.096687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:35.098154 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:35.098171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.099249 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.099255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:35.099473 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:35.100200 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:35.100224 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:35.103169 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:35.107360 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:35.107366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:35.107370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:35.107928 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:35.108381 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:35.108387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:35.108391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:35.108554 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:35.109219 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:35.109226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:35.109406 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 07:09:35.109570 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 07:09:35.109891 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260927 07:09:35.110335 11986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:35.091682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.30.16.124:35371)
0927 07:09:35.091886 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:35.091893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:35.091921 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:09:35.092040 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:35.092048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:35.092143 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:35.092283 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:35.093315 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.094611 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.098322 (+ 3711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:35.099069 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:35.099225 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:35.103403 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:35.103410 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:35.103416 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:35.103457 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:35.107206 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:35.108059 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:35.108065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:35.108070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:35.108239 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:35.108671 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:35.108676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:35.108680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:35.109064 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:35.109221 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:35.109684 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 07:09:35.109927 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 07:09:35.110728 11937 tablet.cc:2369] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:09:35.117087 11909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: Bootstrap starting.
I20260927 07:09:35.121227 11909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:35.122570 11909 log.cc:824] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:35.126128 11909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: No bootstrap required, opened a new log
I20260927 07:09:35.137712 11909 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.138163 11909 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:35.138332 11909 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8d6973619c244d183ffed8de5a86c82, State: Initialized, Role: FOLLOWER
I20260927 07:09:35.139061 11909 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.140041 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.140501 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.141459 11909 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:35.152238 12010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:35.152607 12010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.154304 12010 leader_election.cc:288] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8d6973619c244d183ffed8de5a86c82 (127.30.16.125:44847), dce28f14c8994ae7bd4410e42a07e992 (127.30.16.124:35371)
I20260927 07:09:35.155665 11958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21160ff7c52341b89179695e8820ac14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce28f14c8994ae7bd4410e42a07e992" is_pre_election: true
I20260927 07:09:35.156329 11958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21160ff7c52341b89179695e8820ac14 in term 0.
I20260927 07:09:35.158068 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21160ff7c52341b89179695e8820ac14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d6973619c244d183ffed8de5a86c82" is_pre_election: true
I20260927 07:09:35.158656 11884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21160ff7c52341b89179695e8820ac14 in term 0.
I20260927 07:09:35.159407 11784 leader_election.cc:302] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21160ff7c52341b89179695e8820ac14, dce28f14c8994ae7bd4410e42a07e992; no voters:
I20260927 07:09:35.167454 11842 master_runner.cc:263] No leader master found from master a8d6973619c244d183ffed8de5a86c82
I20260927 07:09:35.167632 11842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:09:35.167932 12010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:09:35.168107 12010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:09:35.168205 12010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:35.174327 12010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
W20260927 07:09:35.174903 12035 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:09:35.175043 12035 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:09:35.177191 11884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21160ff7c52341b89179695e8820ac14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8d6973619c244d183ffed8de5a86c82"
I20260927 07:09:35.177414 11884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:35.177398 11958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21160ff7c52341b89179695e8820ac14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dce28f14c8994ae7bd4410e42a07e992"
I20260927 07:09:35.177630 11958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:35.180835 11884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21160ff7c52341b89179695e8820ac14 in term 1.
I20260927 07:09:35.181231 11786 leader_election.cc:302] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21160ff7c52341b89179695e8820ac14, a8d6973619c244d183ffed8de5a86c82; no voters:
I20260927 07:09:35.180995 11958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21160ff7c52341b89179695e8820ac14 in term 1.
I20260927 07:09:35.182039 12037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:35.182435 12010 leader_election.cc:288] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [CANDIDATE]: Term 1 election: Requested vote from peers a8d6973619c244d183ffed8de5a86c82 (127.30.16.125:44847), dce28f14c8994ae7bd4410e42a07e992 (127.30.16.124:35371)
I20260927 07:09:35.183389 12037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [term 1 LEADER]: Becoming Leader. State: Replica: 21160ff7c52341b89179695e8820ac14, State: Running, Role: LEADER
I20260927 07:09:35.183962 12037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } }
I20260927 07:09:35.199594 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21160ff7c52341b89179695e8820ac14. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.199967 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:35.211794 12040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:35.214407 12040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:35.231437 11958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [term 1 FOLLOWER]: Refusing update from remote peer 21160ff7c52341b89179695e8820ac14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:35.231585 11884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [term 1 FOLLOWER]: Refusing update from remote peer 21160ff7c52341b89179695e8820ac14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:35.232409 12037 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:35.232836 12010 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:35.263432 11996 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21160ff7c52341b89179695e8820ac14. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.263796 11996 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.268787 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21160ff7c52341b89179695e8820ac14. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.269080 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.272233 12043 mvcc.cc:205] Tried to move back new op lower bound from 7333859226533978112 to 7333859226358812672. Current Snapshot: MvccSnapshot[applied={T|T < 7333859226533978112}]
I20260927 07:09:35.284696 11996 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.285012 11996 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.288196 12024 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.288537 12024 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:09:35.292922 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.293274 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:35.293717 12010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21160ff7c52341b89179695e8820ac14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21160ff7c52341b89179695e8820ac14" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33363 } } peers { permanent_uuid: "a8d6973619c244d183ffed8de5a86c82" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 44847 } } peers { permanent_uuid: "dce28f14c8994ae7bd4410e42a07e992" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35371 } } }
I20260927 07:09:35.293984 12010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:35.295240 12040 catalog_manager.cc:1347] Generated new cluster ID: c9db44082b7b4004877cd76a16fa5718
I20260927 07:09:35.295339 12040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:35.333561 12040 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:09:35.341472 12040 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:35.368690 12040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21160ff7c52341b89179695e8820ac14: Generated new TSK 0
I20260927 07:09:35.369977 12040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:09:35.999261 12007 catalog_manager.cc:1259] Loaded cluster ID: c9db44082b7b4004877cd76a16fa5718
I20260927 07:09:35.999361 12007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: loading cluster ID for follower catalog manager: success
I20260927 07:09:36.001868 12007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: acquiring CA information for follower catalog manager: success
I20260927 07:09:36.003924 12007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dce28f14c8994ae7bd4410e42a07e992: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:09:36.177161 12035 catalog_manager.cc:1259] Loaded cluster ID: c9db44082b7b4004877cd76a16fa5718
I20260927 07:09:36.177278 12035 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: loading cluster ID for follower catalog manager: success
I20260927 07:09:36.179865 12035 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: acquiring CA information for follower catalog manager: success
I20260927 07:09:36.181918 12035 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8d6973619c244d183ffed8de5a86c82: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:09:36.509538 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11768
I20260927 07:09:36.510921 11830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:36.627437 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11768
I20260927 07:09:36.649565 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11842
I20260927 07:09:36.650825 11904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:36.758813 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11842
I20260927 07:09:36.779426 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 11916
I20260927 07:09:36.780661 11978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:36.890116 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 11916
2026-09-27T07:09:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4867 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:09:36 dist-test-slave-8vqf krb5kdc[12059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:09:36 dist-test-slave-8vqf krb5kdc[12059](info): set up 2 sockets
Sep 27 07:09:36 dist-test-slave-8vqf krb5kdc[12059](info): commencing operation
krb5kdc: starting...
W20260927 07:09:38.007305 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:09:38 dist-test-slave-8vqf krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:09:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:09:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:09:38.251204 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43187
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:38.591638 12072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:38.592005 12072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:38.592238 12072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:38.600277 12072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:38.600427 12072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:38.600510 12072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:38.600615 12072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:38.600699 12072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:38.611048 12072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:38.612558 12072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:38.613957 12072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:38.626891 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:38.626891 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:38.626932 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:38.627799 12072 server_base.cc:1034] running on GCE node
I20260927 07:09:38.628612 12072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:38.630131 12072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:38.631363 12072 hybrid_clock.cc:648] HybridClock initialized: now 1790492978631307 us; error 50 us; skew 500 ppm
I20260927 07:09:38.631754 12072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:38 dist-test-slave-8vqf krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492978, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:38.635174 12072 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:38.636909 12072 webserver.cc:460] Webserver started at http://127.30.16.126:39759/ using document root <none> and password file <none>
I20260927 07:09:38.637688 12072 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:38.637848 12072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:38.638202 12072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:38.641561 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "9220472237fb472f8beafe35f0e1cab4"
format_stamp: "Formatted at 2026-09-27 07:09:38 on dist-test-slave-8vqf"
server_key: "00fdfafffa69b48b8833f9fd67c9f5a1"
server_key_iv: "73aad3b1172dd3ebb6acbf111188b95e"
server_key_version: "encryptionkey@0"
I20260927 07:09:38.642414 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "9220472237fb472f8beafe35f0e1cab4"
format_stamp: "Formatted at 2026-09-27 07:09:38 on dist-test-slave-8vqf"
server_key: "00fdfafffa69b48b8833f9fd67c9f5a1"
server_key_iv: "73aad3b1172dd3ebb6acbf111188b95e"
server_key_version: "encryptionkey@0"
I20260927 07:09:38.648182 12072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 07:09:38.652038 12087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:38.653466 12072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:09:38.653685 12072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "9220472237fb472f8beafe35f0e1cab4"
format_stamp: "Formatted at 2026-09-27 07:09:38 on dist-test-slave-8vqf"
server_key: "00fdfafffa69b48b8833f9fd67c9f5a1"
server_key_iv: "73aad3b1172dd3ebb6acbf111188b95e"
server_key_version: "encryptionkey@0"
I20260927 07:09:38.653996 12072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:38.667464 12072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:38.672072 12072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:38.672387 12072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:38.692111 12072 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43187
I20260927 07:09:38.692137 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43187 every 8 connection(s)
I20260927 07:09:38.693599 12072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:38.694176 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12072
I20260927 07:09:38.694422 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:09:38.695271 30785 external_mini_cluster.cc:1428] Setting key 2ad7d0d5d0439ea1a219d3d74de3df8b
I20260927 07:09:38.698211 12139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:09:38 dist-test-slave-8vqf krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:38.712198 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4: Bootstrap starting.
I20260927 07:09:38.715524 12139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:38.716572 12139 log.cc:824] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:38.719537 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:38.697214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.30.16.126:43187)
0927 07:09:38.697990 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:38.698002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:38.698044 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:09:38.699820 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:38.699842 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:38.700226 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:38.700537 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:38.702476 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:38.703935 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:38.705830 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:38.706512 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:38.706721 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:38.711842 (+ 5121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:38.711849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:38.711865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:38.711910 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:38.715148 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:38.716012 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:38.716019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:38.716025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:38.716206 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:38.716623 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:38.716629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:38.716633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:38.717384 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:38.717747 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:38.718075 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 07:09:38.718464 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":356,"threads_started":1}
I20260927 07:09:38.719940 12139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4: No bootstrap required, opened a new log
I20260927 07:09:38.723675 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:09:38.728578 12139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } }
I20260927 07:09:38.728973 12139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:38.729081 12139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9220472237fb472f8beafe35f0e1cab4, State: Initialized, Role: FOLLOWER
I20260927 07:09:38.729709 12139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } }
I20260927 07:09:38.729943 12139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:38.730082 12139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:38.730312 12139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:38.732846 12139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } }
I20260927 07:09:38.733372 12139 leader_election.cc:302] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9220472237fb472f8beafe35f0e1cab4; no voters:
I20260927 07:09:38.733963 12139 leader_election.cc:288] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:09:38.734097 12144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:38.734803 12144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [term 1 LEADER]: Becoming Leader. State: Replica: 9220472237fb472f8beafe35f0e1cab4, State: Running, Role: LEADER
I20260927 07:09:38.735343 12144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } }
I20260927 07:09:38.736120 12139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:38.738705 12146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9220472237fb472f8beafe35f0e1cab4. Latest consensus state: current_term: 1 leader_uuid: "9220472237fb472f8beafe35f0e1cab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } } }
I20260927 07:09:38.739114 12146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:38.738443 12145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9220472237fb472f8beafe35f0e1cab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9220472237fb472f8beafe35f0e1cab4" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43187 } } }
I20260927 07:09:38.741683 12145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:38.743577 12152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:38.748234 12152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:38.756735 12152 catalog_manager.cc:1347] Generated new cluster ID: 32b0f74d05af4c5181575b0753202bb6
I20260927 07:09:38.756863 12152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:38.773643 12152 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:09:38.775110 12152 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:38.782972 12152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9220472237fb472f8beafe35f0e1cab4: Generated new TSK 0
I20260927 07:09:38.783825 12152 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:09:38.836372 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12072
I20260927 07:09:38.837733 12134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:38.936380 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12072
2026-09-27T07:09:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2042 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:09:39 dist-test-slave-8vqf krb5kdc[12167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:09:39 dist-test-slave-8vqf krb5kdc[12167](info): set up 2 sockets
Sep 27 07:09:39 dist-test-slave-8vqf krb5kdc[12167](info): commencing operation
krb5kdc: starting...
W20260927 07:09:40.051266 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:09:40 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:09:40.287006 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:40.623957 12180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:40.624342 12180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:40.624511 12180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:40.632112 12180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:40.632256 12180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:40.632375 12180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:40.632470 12180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:40.632557 12180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:40.642683 12180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:40.644130 12180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:40.645486 12180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:40.657805 12185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:40.657882 12188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:40.657842 12186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:40.658893 12180 server_base.cc:1034] running on GCE node
I20260927 07:09:40.659662 12180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:40.661118 12180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:40.662331 12180 hybrid_clock.cc:648] HybridClock initialized: now 1790492980662289 us; error 33 us; skew 500 ppm
I20260927 07:09:40.662730 12180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:40 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:40.671777 12180 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:40.673409 12180 webserver.cc:460] Webserver started at http://127.30.16.126:40917/ using document root <none> and password file <none>
I20260927 07:09:40.674106 12180 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:40.674212 12180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:40.674470 12180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:40.677579 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "fb484acc770d49648841dd59f09b85a5"
format_stamp: "Formatted at 2026-09-27 07:09:40 on dist-test-slave-8vqf"
server_key: "05d0b68547e5ea58f5ef1ea6cdd524e9"
server_key_iv: "be616b14d636573345453adc6c44988c"
server_key_version: "encryptionkey@0"
I20260927 07:09:40.678309 12180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "fb484acc770d49648841dd59f09b85a5"
format_stamp: "Formatted at 2026-09-27 07:09:40 on dist-test-slave-8vqf"
server_key: "05d0b68547e5ea58f5ef1ea6cdd524e9"
server_key_iv: "be616b14d636573345453adc6c44988c"
server_key_version: "encryptionkey@0"
I20260927 07:09:40.683619 12180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 07:09:40.687052 12195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:40.688495 12180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:09:40.688686 12180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "fb484acc770d49648841dd59f09b85a5"
format_stamp: "Formatted at 2026-09-27 07:09:40 on dist-test-slave-8vqf"
server_key: "05d0b68547e5ea58f5ef1ea6cdd524e9"
server_key_iv: "be616b14d636573345453adc6c44988c"
server_key_version: "encryptionkey@0"
I20260927 07:09:40.688907 12180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:40.707281 12180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:40.711946 12180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:40.712268 12180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:40.731511 12180 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43325
I20260927 07:09:40.731545 12246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43325 every 8 connection(s)
I20260927 07:09:40.732944 12180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:40.736861 12247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:40.739938 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12180
I20260927 07:09:40.740149 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:09:40.740835 30785 external_mini_cluster.cc:1428] Setting key 2ffa9caf6dcfc072dfc5348ce7ff0ec3
I20260927 07:09:40.750425 12247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap starting.
I20260927 07:09:40.754014 12247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:40.755100 12247 log.cc:824] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:09:40 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:40.758337 12247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: No bootstrap required, opened a new log
I20260927 07:09:40.766600 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:40.742999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.30.16.126:43325)
0927 07:09:40.743788 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:40.743804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:40.743859 (+ 55us) server_negotiation.cc:407] Connection header received
0927 07:09:40.746039 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:40.746074 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:40.746647 (+ 573us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:40.747099 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:40.748421 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:40.750376 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:40.751839 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:40.752937 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:40.753129 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:40.757686 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:40.757694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:40.757710 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:40.757749 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:40.761798 (+ 4049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:40.762582 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:40.762592 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:40.762601 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:40.762824 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:40.763197 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:40.763202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:40.763206 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:40.763910 (+ 704us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:40.764246 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:40.764647 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 07:09:40.765111 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":336,"threads_started":1}
I20260927 07:09:40.769419 12247 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:40.769872 12247 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:40.769954 12247 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Initialized, Role: FOLLOWER
I20260927 07:09:40.770529 12247 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:40.770778 12247 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:40.770880 12247 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:40.771072 12247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:40.773440 12247 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:40.773875 12247 leader_election.cc:302] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb484acc770d49648841dd59f09b85a5; no voters:
I20260927 07:09:40.774493 12247 leader_election.cc:288] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:09:40.774664 12252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:40.775509 12252 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 LEADER]: Becoming Leader. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Running, Role: LEADER
I20260927 07:09:40.776084 12252 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:40.776933 12247 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:40.779181 12254 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb484acc770d49648841dd59f09b85a5. Latest consensus state: current_term: 1 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:40.779157 12253 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:40.779543 12254 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:40.779543 12253 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:40.783874 12257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:40.789054 12257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:40.798803 12257 catalog_manager.cc:1347] Generated new cluster ID: 99ffd0400b5e41ebb3df24f52c47d334
I20260927 07:09:40.798959 12257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:40.816481 12257 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:09:40.818483 12257 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:40.830171 12257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Generated new TSK 0
I20260927 07:09:40.831494 12257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:09:40.900529 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:41.247797 12273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:41.248085 12273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:41.248252 12273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:41.255918 12273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:41.256023 12273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:41.256197 12273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:09:41.266217 12273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:41.267568 12273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:41.268913 12273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:41.283429 12279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:41.283664 12281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:41.283946 12278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:41.284227 12273 server_base.cc:1034] running on GCE node
I20260927 07:09:41.284893 12273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:41.285981 12273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:41.287227 12273 hybrid_clock.cc:648] HybridClock initialized: now 1790492981287178 us; error 42 us; skew 500 ppm
I20260927 07:09:41.287614 12273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:41 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:41.291141 12273 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:09:41.292809 12273 webserver.cc:460] Webserver started at http://127.30.16.65:45319/ using document root <none> and password file <none>
I20260927 07:09:41.293581 12273 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:41.293735 12273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:41.294060 12273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:41.297616 12273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "a2d5f411db93126fd124f7006244e79c"
server_key_iv: "aa2faf5742451503866ec1a2377d3b05"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.298431 12273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "a2d5f411db93126fd124f7006244e79c"
server_key_iv: "aa2faf5742451503866ec1a2377d3b05"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.304177 12273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:09:41.307952 12288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.309424 12273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:09:41.309716 12273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "a2d5f411db93126fd124f7006244e79c"
server_key_iv: "aa2faf5742451503866ec1a2377d3b05"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.309933 12273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:41.335691 12273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:41.340479 12273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:41.340796 12273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:41.341817 12273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:41.343333 12273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:41.343438 12273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.343590 12273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:41.343665 12273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.379108 12273 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36573
I20260927 07:09:41.379197 12400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36573 every 8 connection(s)
I20260927 07:09:41.380477 12273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:09:41.387912 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12273
I20260927 07:09:41.388209 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:09:41.388885 30785 external_mini_cluster.cc:1428] Setting key 88ffde3bf1b93845fb0edd2a486ecdb6
Sep 27 07:09:41 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:41.401291 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:41.383206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:45099 (local address 127.30.16.126:43325)
0927 07:09:41.383739 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:41.383744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:41.384793 (+ 1049us) server_negotiation.cc:407] Connection header received
0927 07:09:41.386746 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:41.386751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:41.386845 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:41.386962 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:41.388626 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.389411 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.391085 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.391475 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.391620 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:41.395449 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:41.395454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:41.395457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:41.395489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:41.398140 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:41.398981 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:41.398987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:41.398989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:41.399106 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:41.399539 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:41.399550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:41.399553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:41.399813 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:41.399972 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:41.400695 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 07:09:41.400927 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":287,"threads_started":1}
I20260927 07:09:41.402297 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:41.383811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:45099)
0927 07:09:41.384608 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:41.384675 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:41.386409 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:41.387216 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:41.387225 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:41.387747 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:41.388454 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:41.388473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.389584 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.389589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:41.390926 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:41.390938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.391710 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.391719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:41.391979 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:41.392694 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:41.392750 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:41.395228 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:41.398291 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:41.398300 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:41.398318 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:41.398847 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:41.399218 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:41.399223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:41.399227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:41.399413 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:41.399957 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:41.399964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:41.400319 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 07:09:41.400587 (+ 268us) client_negotiation.cc:240] Negotiation successful
0927 07:09:41.401149 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":342,"threads_started":1}
I20260927 07:09:41.404157 12401 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:41.404500 12401 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:41.405441 12401 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:41.407989 12212 ts_manager.cc:194] Registered new tserver with Master: e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573)
I20260927 07:09:41.409528 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:45099
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:09:41.442378 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:41.785522 12407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:41.785808 12407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:41.785903 12407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:41.793352 12407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:41.793467 12407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:41.793582 12407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:09:41.803435 12407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:41.804803 12407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:41.806095 12407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:41.820266 12415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:41.820286 12413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:41.820501 12412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:41.820984 12407 server_base.cc:1034] running on GCE node
I20260927 07:09:41.821633 12407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:41.822728 12407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:41.823987 12407 hybrid_clock.cc:648] HybridClock initialized: now 1790492981823935 us; error 40 us; skew 500 ppm
I20260927 07:09:41.824405 12407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:41 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:41.827746 12407 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:09:41.829322 12407 webserver.cc:460] Webserver started at http://127.30.16.66:43145/ using document root <none> and password file <none>
I20260927 07:09:41.830080 12407 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:41.830250 12407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:41.830693 12407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:41.833792 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "ed03c78e28914c45b50f7ef20fef8ac5"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "e6e9a2fffc73c148f7728c5d94db3e04"
server_key_iv: "41d92eeb59873a3f1fef001b198d8351"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.834668 12407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "ed03c78e28914c45b50f7ef20fef8ac5"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "e6e9a2fffc73c148f7728c5d94db3e04"
server_key_iv: "41d92eeb59873a3f1fef001b198d8351"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.840158 12407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 07:09:41.843653 12422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.845216 12407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:41.845400 12407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "ed03c78e28914c45b50f7ef20fef8ac5"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "e6e9a2fffc73c148f7728c5d94db3e04"
server_key_iv: "41d92eeb59873a3f1fef001b198d8351"
server_key_version: "encryptionkey@0"
I20260927 07:09:41.845619 12407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:41.863103 12407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:41.867441 12407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:41.867759 12407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:41.868737 12407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:41.870124 12407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:41.870235 12407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.870401 12407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:41.870488 12407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:41.902595 12407 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:42521
I20260927 07:09:41.902684 12534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:42521 every 8 connection(s)
I20260927 07:09:41.904059 12407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:09:41.906243 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12407
I20260927 07:09:41.906440 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:09:41.907122 30785 external_mini_cluster.cc:1428] Setting key ccc388d5d659eb62dd58a677bef1142e
Sep 27 07:09:41 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492981, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:41.928396 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:41.906847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:49491 (local address 127.30.16.126:43325)
0927 07:09:41.907362 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:41.907369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:41.908504 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 07:09:41.910365 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:41.910373 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:41.910466 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:41.910631 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:41.912325 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.913583 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.915407 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.916150 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.916249 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:41.920517 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:41.920526 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:41.920534 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:41.920585 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:41.924487 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:41.925509 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:41.925518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:41.925525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:41.925703 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:41.926263 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:41.926272 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:41.926278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:41.926745 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:41.926905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:41.927754 (+ 849us) server_negotiation.cc:299] Negotiation successful
0927 07:09:41.928005 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260927 07:09:41.929486 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:41.907504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:49491)
0927 07:09:41.908339 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:41.908384 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:41.910093 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:41.910897 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:41.910906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:41.911527 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:41.912140 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:41.912157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.913745 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.913749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:41.915267 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:41.915280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:41.916403 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:41.916413 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:41.916741 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:41.917528 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:41.917558 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:41.920223 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:41.924696 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:41.924706 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:41.924723 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:41.925346 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:41.925857 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:41.925861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:41.925863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:41.926118 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:41.926895 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:41.926901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:41.927339 (+ 438us) client_negotiation.cc:769] Sending connection context
0927 07:09:41.927657 (+ 318us) client_negotiation.cc:240] Negotiation successful
0927 07:09:41.928281 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":350,"threads_started":1}
I20260927 07:09:41.931154 12535 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:41.931483 12535 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:09:41.932273 12535 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:41.933828 12212 ts_manager.cc:194] Registered new tserver with Master: ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:09:41.934763 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:49491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:09:41.961997 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:42.295635 12541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:42.295919 12541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:42.296113 12541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:42.303519 12541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:42.303642 12541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:42.303881 12541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:09:42.314016 12541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:42.315451 12541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:42.316761 12541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:42.330505 12547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:42.330621 12549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:42.330506 12546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:42.331725 12541 server_base.cc:1034] running on GCE node
I20260927 07:09:42.332427 12541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:42.333485 12541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:42.334707 12541 hybrid_clock.cc:648] HybridClock initialized: now 1790492982334636 us; error 61 us; skew 500 ppm
I20260927 07:09:42.335069 12541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:42 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492982, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:42.338317 12541 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:09:42.340152 12541 webserver.cc:460] Webserver started at http://127.30.16.67:46055/ using document root <none> and password file <none>
I20260927 07:09:42.340992 12541 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:42.341163 12541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:42.341511 12541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:09:42.344908 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "81bcf4f44471488fac08b5a1829dce9e"
format_stamp: "Formatted at 2026-09-27 07:09:42 on dist-test-slave-8vqf"
server_key: "592e0dc22363c3336f4ad7eeb472fe00"
server_key_iv: "694b9b9c66936c496e9b3adb531a6b1e"
server_key_version: "encryptionkey@0"
I20260927 07:09:42.345824 12541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "81bcf4f44471488fac08b5a1829dce9e"
format_stamp: "Formatted at 2026-09-27 07:09:42 on dist-test-slave-8vqf"
server_key: "592e0dc22363c3336f4ad7eeb472fe00"
server_key_iv: "694b9b9c66936c496e9b3adb531a6b1e"
server_key_version: "encryptionkey@0"
I20260927 07:09:42.352100 12541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260927 07:09:42.355754 12556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:42.357108 12541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:42.357296 12541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "81bcf4f44471488fac08b5a1829dce9e"
format_stamp: "Formatted at 2026-09-27 07:09:42 on dist-test-slave-8vqf"
server_key: "592e0dc22363c3336f4ad7eeb472fe00"
server_key_iv: "694b9b9c66936c496e9b3adb531a6b1e"
server_key_version: "encryptionkey@0"
I20260927 07:09:42.357508 12541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:42.375131 12541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:42.379877 12541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:42.380183 12541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:42.381183 12541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:42.382723 12541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:42.382840 12541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:42.383000 12541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:42.383071 12541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:42.413223 12401 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:42.416203 12541 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:42013
I20260927 07:09:42.416290 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:42013 every 8 connection(s)
I20260927 07:09:42.417562 12541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:09:42.426930 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12541
I20260927 07:09:42.427158 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:09:42.427627 30785 external_mini_cluster.cc:1428] Setting key 730427e80949e9194560fdc49e58d42a
Sep 27 07:09:42 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492982, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:42.438473 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.420220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:44259 (local address 127.30.16.126:43325)
0927 07:09:42.420393 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.420398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.421663 (+ 1265us) server_negotiation.cc:407] Connection header received
0927 07:09:42.423585 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.423590 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.423681 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.423774 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:42.425409 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.426189 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.428052 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.428505 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.428610 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.432646 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:42.432651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:42.432654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:42.432691 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:42.435589 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.436310 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.436314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.436316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.436433 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.436815 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.436818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.436820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.437084 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:42.437206 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.438046 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.438234 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 07:09:42.439604 12671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.420703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:44259)
0927 07:09:42.421475 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.421526 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.423286 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.424010 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.424021 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.424620 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:42.425234 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.425252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.426342 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.426346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.427899 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.427912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.428756 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.428763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.429089 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.429873 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:42.429896 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:42.432436 (+ 2540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:42.435725 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:42.435733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:42.435748 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:42.436199 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:42.436522 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:42.436525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:42.436527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:42.436718 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:42.437188 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:42.437194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:42.437625 (+ 431us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.437968 (+ 343us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.438407 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":325,"threads_started":1}
I20260927 07:09:42.441098 12669 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:42.441555 12669 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:42.442602 12669 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:42.444294 12212 ts_manager.cc:194] Registered new tserver with Master: 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:42.445109 12212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:44259
I20260927 07:09:42.456552 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:09:42.474738 12537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.461262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39382 (local address 127.30.16.126:43325)
0927 07:09:42.461418 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.461423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.461911 (+ 488us) server_negotiation.cc:407] Connection header received
0927 07:09:42.462123 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.462127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.462195 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.462298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:42.463745 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.464476 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.466244 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.466735 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.466835 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.469236 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:42.469241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:42.469244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:42.469278 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:42.472199 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.472863 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.472868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.472870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.473013 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.473370 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.473374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.473376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.473725 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:42.473922 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.474176 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.474343 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:09:42.479182 12212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:09:42.486229 12212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:09:42.510576 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.498865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56160 (local address 127.30.16.66:42521)
0927 07:09:42.499506 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.499516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.500090 (+ 574us) server_negotiation.cc:407] Connection header received
0927 07:09:42.500408 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.500419 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.500698 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.500877 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:42.501938 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.503780 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.507289 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.508881 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.509025 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.509236 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.509833 (+ 597us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.510169 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":347,"threads_started":1}
I20260927 07:09:42.511059 12683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.499326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54534)
0927 07:09:42.499918 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.499962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.500148 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.501584 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.501590 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.501616 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:42.502063 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.502078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.504435 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.504440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.506701 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.506719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.508696 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.508704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.509057 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.509139 (+ 82us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.509252 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.509470 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":1,"thread_start_us":761,"threads_started":1}
I20260927 07:09:42.511278 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.499928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.30.16.65:36573)
0927 07:09:42.500960 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.500968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.500995 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:09:42.501092 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.501102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.501346 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.502271 (+ 925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:42.502499 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.504281 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.506919 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.508445 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.508590 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.510381 (+ 1791us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.510510 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.510789 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":1186,"threads_started":1}
I20260927 07:09:42.511876 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.498789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42521 (local address 127.0.0.1:56160)
0927 07:09:42.499918 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.499962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.500176 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.501200 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.501207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.501235 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:42.501709 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.501727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.503976 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.503982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.507105 (+ 3123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.507119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.509388 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.509394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.509589 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.509636 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.511138 (+ 1502us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.511444 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":560,"thread_start_us":772,"threads_started":1}
I20260927 07:09:42.515882 12470 tablet_service.cc:1467] Processing CreateTablet for tablet 0f326f7903b640ff82d10087fcd4337b (DEFAULT_TABLE table=test-table [id=3723c49bcaec4c63899431af6926e0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:42.515882 12336 tablet_service.cc:1467] Processing CreateTablet for tablet 0f326f7903b640ff82d10087fcd4337b (DEFAULT_TABLE table=test-table [id=3723c49bcaec4c63899431af6926e0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:42.517347 12470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f326f7903b640ff82d10087fcd4337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:42.517620 12336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f326f7903b640ff82d10087fcd4337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:42.518491 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.500426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:42013 (local address 127.0.0.1:33018)
0927 07:09:42.507378 (+ 6952us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.507394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.507541 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.508079 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.508083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.508108 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:42.509135 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.509148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.513515 (+ 4367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.513520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.515637 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.515654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.517717 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.517724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.517890 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.517925 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.518036 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.518164 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6842,"thread_start_us":6682,"threads_started":1}
I20260927 07:09:42.519258 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.500757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33018 (local address 127.30.16.67:42013)
0927 07:09:42.506993 (+ 6236us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.507003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.507506 (+ 503us) server_negotiation.cc:407] Connection header received
0927 07:09:42.507714 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.507720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.507902 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.508091 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:42.509341 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.513289 (+ 3948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.515829 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.517211 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.517344 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.517583 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.518757 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.518924 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6043,"thread_start_us":5886,"threads_started":1}
I20260927 07:09:42.523077 12604 tablet_service.cc:1467] Processing CreateTablet for tablet 0f326f7903b640ff82d10087fcd4337b (DEFAULT_TABLE table=test-table [id=3723c49bcaec4c63899431af6926e0d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:42.524621 12604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f326f7903b640ff82d10087fcd4337b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:42.530313 12687 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Bootstrap starting.
I20260927 07:09:42.532298 12688 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Bootstrap starting.
I20260927 07:09:42.535148 12687 tablet_bootstrap.cc:654] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:42.535892 12688 tablet_bootstrap.cc:654] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:42.536497 12687 log.cc:824] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:42.537153 12688 log.cc:824] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:42.541062 12688 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: No bootstrap required, opened a new log
I20260927 07:09:42.541069 12687 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: No bootstrap required, opened a new log
I20260927 07:09:42.541364 12687 ts_tablet_manager.cc:1397] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 07:09:42.541550 12688 ts_tablet_manager.cc:1397] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 07:09:42.547350 12691 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Bootstrap starting.
I20260927 07:09:42.550791 12688 raft_consensus.cc:348] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.551276 12688 raft_consensus.cc:374] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:42.551383 12688 raft_consensus.cc:729] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed03c78e28914c45b50f7ef20fef8ac5, State: Initialized, Role: FOLLOWER
I20260927 07:09:42.551744 12691 tablet_bootstrap.cc:654] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:42.552057 12688 consensus_queue.cc:260] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.552668 12687 raft_consensus.cc:348] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.553109 12691 log.cc:824] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:42.553336 12535 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:42.553957 12688 ts_tablet_manager.cc:1428] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 07:09:42.553123 12687 raft_consensus.cc:374] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:42.554776 12687 raft_consensus.cc:729] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c8b5f362343fd98fb614ba9df3ae0, State: Initialized, Role: FOLLOWER
I20260927 07:09:42.555420 12687 consensus_queue.cc:260] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.559526 12687 ts_tablet_manager.cc:1428] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260927 07:09:42.563375 12691 tablet_bootstrap.cc:492] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: No bootstrap required, opened a new log
I20260927 07:09:42.563686 12691 ts_tablet_manager.cc:1397] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20260927 07:09:42.571393 12691 raft_consensus.cc:348] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.571812 12691 raft_consensus.cc:374] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:42.571920 12691 raft_consensus.cc:729] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81bcf4f44471488fac08b5a1829dce9e, State: Initialized, Role: FOLLOWER
I20260927 07:09:42.572492 12691 consensus_queue.cc:260] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.573560 12669 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:42.574216 12691 ts_tablet_manager.cc:1428] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
W20260927 07:09:42.632972 12402 tablet.cc:2369] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:09:42.656286 12536 tablet.cc:2369] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:09:42.657734 12692 raft_consensus.cc:482] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:42.658015 12692 raft_consensus.cc:504] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.659997 12692 leader_election.cc:288] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573), 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:42.668921 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.660461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:52691 (local address 127.30.16.65:36573)
0927 07:09:42.660661 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.660668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.661847 (+ 1179us) server_negotiation.cc:407] Connection header received
0927 07:09:42.661982 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.661986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.662044 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.662113 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:42.662767 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.663784 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.666336 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.667653 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.667822 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.667900 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.668525 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.668684 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:09:42.668918 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.660460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:39521 (local address 127.30.16.67:42013)
0927 07:09:42.660661 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.660668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.661202 (+ 534us) server_negotiation.cc:407] Connection header received
0927 07:09:42.661589 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.661594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.661661 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.661740 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:42.662395 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.663575 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.666336 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.667663 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.667822 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.667900 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.668525 (+ 625us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.668654 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 07:09:42.669250 12670 tablet.cc:2369] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:09:42.669500 12697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.660254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:42013 (local address 127.30.16.66:39521)
0927 07:09:42.661088 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.661104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.661345 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.661787 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.661793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.661820 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:42.662203 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.662215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.663708 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.663711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.666158 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.666171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.667967 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.667974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.668242 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.668391 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.668499 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.668634 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":141,"thread_start_us":1117,"threads_started":1}
I20260927 07:09:42.669698 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.660267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.30.16.66:52691)
0927 07:09:42.661698 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.661731 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:42.661839 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:42.662167 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:42.662172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:42.662201 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:42.662530 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.662538 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.663913 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.663918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.666157 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:42.666171 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.667969 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.667974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:42.668239 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.668381 (+ 142us) client_negotiation.cc:769] Sending connection context
0927 07:09:42.668486 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:09:42.668634 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":179,"thread_start_us":532,"threads_started":1}
I20260927 07:09:42.670490 12624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f326f7903b640ff82d10087fcd4337b" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bcf4f44471488fac08b5a1829dce9e" is_pre_election: true
I20260927 07:09:42.670755 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f326f7903b640ff82d10087fcd4337b" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" is_pre_election: true
I20260927 07:09:42.671069 12624 raft_consensus.cc:2454] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 0.
I20260927 07:09:42.671315 12356 raft_consensus.cc:2454] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 0.
I20260927 07:09:42.671754 12423 leader_election.cc:302] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5; no voters:
I20260927 07:09:42.672257 12692 raft_consensus.cc:2781] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:09:42.672405 12692 raft_consensus.cc:482] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:09:42.672546 12692 raft_consensus.cc:3037] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:42.675050 12692 raft_consensus.cc:504] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.675706 12692 leader_election.cc:288] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 election: Requested vote from peers e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573), 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:42.676244 12356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f326f7903b640ff82d10087fcd4337b" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
I20260927 07:09:42.676242 12624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f326f7903b640ff82d10087fcd4337b" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bcf4f44471488fac08b5a1829dce9e"
I20260927 07:09:42.676465 12356 raft_consensus.cc:3037] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:42.676465 12624 raft_consensus.cc:3037] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:42.679726 12356 raft_consensus.cc:2454] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 1.
I20260927 07:09:42.679726 12624 raft_consensus.cc:2454] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 1.
I20260927 07:09:42.680230 12426 leader_election.cc:302] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0, ed03c78e28914c45b50f7ef20fef8ac5; no voters:
I20260927 07:09:42.680581 12692 raft_consensus.cc:2781] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:42.681054 12692 raft_consensus.cc:686] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 LEADER]: Becoming Leader. State: Replica: ed03c78e28914c45b50f7ef20fef8ac5, State: Running, Role: LEADER
I20260927 07:09:42.681552 12692 consensus_queue.cc:237] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } }
I20260927 07:09:42.686479 12212 catalog_manager.cc:5495] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } health_report { overall_health: UNKNOWN } } }
I20260927 07:09:42.707532 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.701960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.30.16.66:42521)
0927 07:09:42.702227 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.702235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.702255 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:09:42.702353 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.702358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.702434 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.702644 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:09:42.703036 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.703740 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.705278 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.705694 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.705835 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.706306 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:09:42.706920 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:09:42.707112 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.707160 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.707239 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260927 07:09:42.716202 12624 raft_consensus.cc:1261] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Refusing update from remote peer ed03c78e28914c45b50f7ef20fef8ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:42.716202 12356 raft_consensus.cc:1261] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Refusing update from remote peer ed03c78e28914c45b50f7ef20fef8ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:42.717360 12698 consensus_queue.cc:1035] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:42.717736 12692 consensus_queue.cc:1035] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:42.729403 12701 mvcc.cc:205] Tried to move back new op lower bound from 7333859257194393600 to 7333859257067401216. Current Snapshot: MvccSnapshot[applied={T|T < 7333859257194393600}]
I20260927 07:09:42.746599 12212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:42.746878 12212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:42.751315 12212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Sending DeleteTablet for 3 replicas of tablet 0f326f7903b640ff82d10087fcd4337b
I20260927 07:09:42.752419 12470 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f326f7903b640ff82d10087fcd4337b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:42 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56160
I20260927 07:09:42.752463 12336 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f326f7903b640ff82d10087fcd4337b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:42 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:54534
I20260927 07:09:42.753320 12711 tablet_replica.cc:330] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: stopping tablet replica
I20260927 07:09:42.753830 12711 raft_consensus.cc:2229] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:09:42.754617 12711 raft_consensus.cc:2258] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:42.755278 12604 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f326f7903b640ff82d10087fcd4337b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:42 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33018
I20260927 07:09:42.755831 12712 tablet_replica.cc:330] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: stopping tablet replica
I20260927 07:09:42.755959 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12273
I20260927 07:09:42.756275 12712 raft_consensus.cc:2229] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:42.758944 12712 raft_consensus.cc:2258] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:42.761337 12711 ts_tablet_manager.cc:1905] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:42.766011 12713 tablet_replica.cc:330] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: stopping tablet replica
I20260927 07:09:42.766446 12713 raft_consensus.cc:2229] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:42.767041 12713 raft_consensus.cc:2258] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 07:09:42 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:09:42.769347 12713 ts_tablet_manager.cc:1905] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:42.772367 12711 ts_tablet_manager.cc:1918] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:42.772644 12711 log.cc:1196] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/0f326f7903b640ff82d10087fcd4337b
I20260927 07:09:42.773298 12711 ts_tablet_manager.cc:1939] T 0f326f7903b640ff82d10087fcd4337b P ed03c78e28914c45b50f7ef20fef8ac5: Deleting consensus metadata
I20260927 07:09:42.773363 12712 ts_tablet_manager.cc:1905] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:42.775377 12196 catalog_manager.cc:4841] TS ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): tablet 0f326f7903b640ff82d10087fcd4337b (table test-table [id=3723c49bcaec4c63899431af6926e0d4]) successfully deleted
I20260927 07:09:42.778667 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.757594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54536 (local address 127.30.16.65:36573)
0927 07:09:42.757814 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.757822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.757847 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:09:42.757921 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.757929 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.758027 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.758158 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:42.760282 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.761444 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.763523 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.764165 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.764270 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.770666 (+ 6396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:42.770687 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:42.770706 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:42.770756 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:42.774928 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.775946 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.775956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.775960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.776147 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.776603 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.776608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.776612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.777173 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:42.777491 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.777835 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.778231 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:09:42.779763 12396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:42.782691 12713 ts_tablet_manager.cc:1918] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:42.782873 12713 log.cc:1196] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/0f326f7903b640ff82d10087fcd4337b
I20260927 07:09:42.783425 12713 ts_tablet_manager.cc:1939] T 0f326f7903b640ff82d10087fcd4337b P 81bcf4f44471488fac08b5a1829dce9e: Deleting consensus metadata
I20260927 07:09:42.785383 12196 catalog_manager.cc:4841] TS 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): tablet 0f326f7903b640ff82d10087fcd4337b (table test-table [id=3723c49bcaec4c63899431af6926e0d4]) successfully deleted
I20260927 07:09:42.872783 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12273
I20260927 07:09:42.891801 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12407
W20260927 07:09:42.891834 12199 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.16.65:36573 (error 108)
I20260927 07:09:42.895093 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.894596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54544)
0927 07:09:42.894785 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.894912 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 07:09:42.895390 12199 catalog_manager.cc:4584] TS e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0f326f7903b640ff82d10087fcd4337b: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Sep 27 07:09:42 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:09:42.912765 12682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.892737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56186 (local address 127.30.16.66:42521)
0927 07:09:42.893008 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:42.893014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:42.893325 (+ 311us) server_negotiation.cc:407] Connection header received
0927 07:09:42.893580 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:42.893591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:42.893735 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:42.893858 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:42.895896 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.897124 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.898468 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:42.899409 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:42.899540 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:42.904630 (+ 5090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:42.904642 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:42.904657 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:42.904717 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:42.909199 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.910114 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.910123 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.910127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.910298 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:42.910753 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:42.910760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:42.910763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:42.911246 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:42.911573 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:42.911901 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 07:09:42.912155 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:09:42.913515 12530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:42.949218 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.948550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54546)
0927 07:09:42.948823 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.948939 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260927 07:09:42.950335 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:42.949926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54548)
0927 07:09:42.950089 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:42.950172 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 07:09:43.011338 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.010760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54560)
0927 07:09:43.011012 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:43.011122 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260927 07:09:43.012543 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.012119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54568)
0927 07:09:43.012304 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:43.012399 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":3}
I20260927 07:09:43.032562 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12407
I20260927 07:09:43.048266 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12541
Sep 27 07:09:43 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:09:43.063783 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.049134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.30.16.67:42013)
0927 07:09:43.049330 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:43.049334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:43.049429 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:09:43.049594 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:43.049599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:43.049665 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:43.049758 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:43.051195 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:43.051960 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:43.053228 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:43.053651 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:43.053751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:43.058251 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:43.058267 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:43.058283 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:43.058316 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:43.061102 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:43.061816 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:43.061822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:43.061825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:43.061964 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:43.062286 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:43.062289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:43.062291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:43.062870 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:43.063102 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:43.063358 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 07:09:43.063539 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 07:09:43.064393 12664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:43.093114 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.092594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54574)
0927 07:09:43.092815 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:43.092924 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 07:09:43.094344 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.093916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54580)
0927 07:09:43.094124 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:43.094214 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 07:09:43.154865 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12541
I20260927 07:09:43.172014 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12180
I20260927 07:09:43.173175 12242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:43.285970 12685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:43.285184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54592)
0927 07:09:43.285592 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:43.285722 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260927 07:09:43.286229 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12180
I20260927 07:09:43.307837 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--webserver_interface=127.30.16.126
--webserver_port=40917
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:43.649024 12720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:43.649405 12720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:43.649590 12720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:43.657145 12720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:43.657325 12720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:43.657433 12720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:43.657536 12720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:43.657624 12720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:43.668094 12720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=40917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:43.669566 12720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:43.671018 12720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:43.697732 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:43.697767 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:43.697832 12720 server_base.cc:1034] running on GCE node
W20260927 07:09:43.697759 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:43.699183 12720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:43.701002 12720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:43.702255 12720 hybrid_clock.cc:648] HybridClock initialized: now 1790492983702184 us; error 61 us; skew 500 ppm
I20260927 07:09:43.702741 12720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:43 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492983, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:43.706362 12720 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:43.708092 12720 webserver.cc:460] Webserver started at http://127.30.16.126:40917/ using document root <none> and password file <none>
I20260927 07:09:43.709290 12720 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:43.709455 12720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:43.715837 12720 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 07:09:43.719009 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:43.720496 12720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:43.720721 12720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "fb484acc770d49648841dd59f09b85a5"
format_stamp: "Formatted at 2026-09-27 07:09:40 on dist-test-slave-8vqf"
server_key: "05d0b68547e5ea58f5ef1ea6cdd524e9"
server_key_iv: "be616b14d636573345453adc6c44988c"
server_key_version: "encryptionkey@0"
I20260927 07:09:43.721863 12720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:43.752625 12720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:43.757221 12720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:43.757529 12720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:43.778049 12720 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43325
I20260927 07:09:43.778070 12787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43325 every 8 connection(s)
I20260927 07:09:43.779744 12720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:43.782112 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12720
I20260927 07:09:43.783159 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:36573
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=45319
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:43.789997 12788 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:09:43.794219 12788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap starting.
I20260927 07:09:43.815505 12788 log.cc:824] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:43.829142 12788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:09:43.829764 12788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap complete.
I20260927 07:09:43.840072 12788 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:43.840988 12788 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Initialized, Role: FOLLOWER
I20260927 07:09:43.841715 12788 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:43.841974 12788 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:43.842123 12788 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:43.842311 12788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:09:43.846491 12788 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:43.847174 12788 leader_election.cc:302] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb484acc770d49648841dd59f09b85a5; no voters:
I20260927 07:09:43.847863 12788 leader_election.cc:288] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:09:43.848017 12792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:09:43.849275 12792 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 LEADER]: Becoming Leader. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Running, Role: LEADER
I20260927 07:09:43.849855 12792 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:43.850302 12788 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:43.852779 12794 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb484acc770d49648841dd59f09b85a5. Latest consensus state: current_term: 2 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:43.852865 12793 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:43.853219 12794 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:43.853291 12793 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:43.864017 12798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:43.874511 12798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:43.875859 12809 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:09:43.876338 12798 catalog_manager.cc:1259] Loaded cluster ID: 99ffd0400b5e41ebb3df24f52c47d334
I20260927 07:09:43.876855 12798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:09:43.879281 12798 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 07:09:43.890836 12798 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 07:09:43.901903 12798 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:09:43.903450 12798 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Loaded TSK: 0
I20260927 07:09:43.904603 12798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:09:44.168622 12790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:44.168931 12790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:44.169080 12790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:44.176817 12790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:44.176940 12790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:44.177117 12790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:09:44.187351 12790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:36573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=45319
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:44.188750 12790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:44.190264 12790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:44.204674 12815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:44.204638 12818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:44.205041 12816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:44.205333 12790 server_base.cc:1034] running on GCE node
I20260927 07:09:44.206061 12790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:44.207173 12790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:44.208393 12790 hybrid_clock.cc:648] HybridClock initialized: now 1790492984208325 us; error 65 us; skew 500 ppm
I20260927 07:09:44.208734 12790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:44 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:44.212177 12790 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:09:44.213822 12790 webserver.cc:460] Webserver started at http://127.30.16.65:45319/ using document root <none> and password file <none>
I20260927 07:09:44.214624 12790 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:44.214779 12790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:44.220853 12790 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 07:09:44.224123 12825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:44.225751 12790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:09:44.225916 12790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "a2d5f411db93126fd124f7006244e79c"
server_key_iv: "aa2faf5742451503866ec1a2377d3b05"
server_key_version: "encryptionkey@0"
I20260927 07:09:44.227015 12790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:44.250903 12790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:44.255399 12790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:44.255681 12790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:44.256718 12790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:44.258838 12832 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260927 07:09:44.266371 12832 ts_tablet_manager.cc:1852] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 07:09:44.267580 12832 ts_tablet_manager.cc:1905] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:44.275197 12832 ts_tablet_manager.cc:1918] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:44.275379 12832 log.cc:1196] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/0f326f7903b640ff82d10087fcd4337b
I20260927 07:09:44.275805 12832 ts_tablet_manager.cc:1939] T 0f326f7903b640ff82d10087fcd4337b P e49c8b5f362343fd98fb614ba9df3ae0: Deleting consensus metadata
I20260927 07:09:44.276167 12790 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 07:09:44.276296 12790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260927 07:09:44.276450 12790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:44.276517 12790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:44.310959 12790 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36573
I20260927 07:09:44.311023 12938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36573 every 8 connection(s)
I20260927 07:09:44.312345 12790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:09:44.320623 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12790
I20260927 07:09:44.321237 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:42521
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=43145
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:44 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:44.339205 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:44.315498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:51995 (local address 127.30.16.126:43325)
0927 07:09:44.316196 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:44.316209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:44.316755 (+ 546us) server_negotiation.cc:407] Connection header received
0927 07:09:44.319131 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:44.319147 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:44.319531 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:44.319854 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:44.321623 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.323787 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.325614 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.326823 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.327036 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:44.331190 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:44.331217 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:44.331243 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:44.331306 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:44.334333 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:44.335191 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:44.335205 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:44.335210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:44.335378 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:44.335832 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:44.335836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:44.335838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:44.336347 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:44.336565 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:44.337389 (+ 824us) server_negotiation.cc:299] Negotiation successful
0927 07:09:44.337796 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":278,"threads_started":1}
I20260927 07:09:44.339205 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:44.315737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:51995)
0927 07:09:44.316563 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:44.316617 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:44.318498 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:44.320134 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:44.320143 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:44.320725 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:44.321366 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:44.321394 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.324011 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.324015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:44.325448 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:44.325462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.327091 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.327097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:44.327320 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:44.328060 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:44.328092 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:44.330793 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:44.334507 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:44.334516 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:44.334533 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:44.335039 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:44.335499 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:44.335507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:44.335511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:44.335704 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:44.336493 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:44.336501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:44.336863 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 07:09:44.337168 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 07:09:44.337763 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":340,"threads_started":1}
I20260927 07:09:44.342685 12939 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:44.343078 12939 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:44.343812 12939 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:44.346257 12753 ts_manager.cc:194] Registered new tserver with Master: e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573)
I20260927 07:09:44.348480 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:51995
W20260927 07:09:44.673027 12943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:44.673348 12943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:44.673457 12943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:44.680883 12943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:44.681007 12943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:44.681133 12943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:09:44.691146 12943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:42521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=43145
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:44.692448 12943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:44.693743 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:44.707072 12949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:44.707232 12951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:44.707285 12948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:44.708449 12943 server_base.cc:1034] running on GCE node
I20260927 07:09:44.709131 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:44.710210 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:44.711434 12943 hybrid_clock.cc:648] HybridClock initialized: now 1790492984711370 us; error 53 us; skew 500 ppm
I20260927 07:09:44.711759 12943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:44 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:44.714907 12943 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:09:44.716408 12943 webserver.cc:460] Webserver started at http://127.30.16.66:43145/ using document root <none> and password file <none>
I20260927 07:09:44.717065 12943 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:44.717192 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:44.722677 12943 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:44.725682 12958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:44.727461 12943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:09:44.727644 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "ed03c78e28914c45b50f7ef20fef8ac5"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "e6e9a2fffc73c148f7728c5d94db3e04"
server_key_iv: "41d92eeb59873a3f1fef001b198d8351"
server_key_version: "encryptionkey@0"
I20260927 07:09:44.728669 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:44.747769 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:44.752430 12943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:44.752723 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:44.753747 12943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:44.755165 12943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:44.755257 12943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:44.755380 12943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:44.755455 12943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:44.790153 12943 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:42521
I20260927 07:09:44.790215 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:42521 every 8 connection(s)
I20260927 07:09:44.791640 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:09:44.796590 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 12943
I20260927 07:09:44.797369 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:42013
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=46055
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:09:44 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492984, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:44.815596 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:44.794368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:38499 (local address 127.30.16.126:43325)
0927 07:09:44.794597 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:44.794603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:44.795903 (+ 1300us) server_negotiation.cc:407] Connection header received
0927 07:09:44.798826 (+ 2923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:44.798831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:44.798994 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:44.799126 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:44.801420 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.802194 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.804866 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.805555 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.805706 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:44.809412 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:44.809417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:44.809421 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:44.809464 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:44.812195 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:44.813001 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:44.813005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:44.813007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:44.813141 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:44.813602 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:44.813606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:44.813608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:44.813883 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:44.814048 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:44.814963 (+ 915us) server_negotiation.cc:299] Negotiation successful
0927 07:09:44.815213 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:09:44.816690 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:44.794902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:38499)
0927 07:09:44.795719 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:44.795767 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:44.798453 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:44.799408 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:44.799423 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:44.800290 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:44.801188 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:44.801218 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.802377 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.802386 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:44.804678 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:44.804698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:44.805774 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:44.805779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:44.806015 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:44.806779 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:44.806803 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:44.809165 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:44.812360 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:44.812371 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:44.812390 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:44.812852 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:44.813253 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:44.813259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:44.813262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:44.813476 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:44.814020 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:44.814029 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:44.814503 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 07:09:44.814848 (+ 345us) client_negotiation.cc:240] Negotiation successful
0927 07:09:44.815454 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":293,"threads_started":1}
I20260927 07:09:44.818481 13071 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:44.818863 13071 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:44.819682 13071 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:44.821446 12753 ts_manager.cc:194] Registered new tserver with Master: ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
I20260927 07:09:44.822803 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:38499
W20260927 07:09:45.137801 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:45.138096 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:45.138191 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:45.145608 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:45.145725 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:45.145838 13074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:09:45.155781 13074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:42013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=46055
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:45.157088 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:45.158459 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:45.172545 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:45.172623 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:45.172595 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:45.173621 13074 server_base.cc:1034] running on GCE node
I20260927 07:09:45.174314 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:45.175401 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:45.176642 13074 hybrid_clock.cc:648] HybridClock initialized: now 1790492985176571 us; error 60 us; skew 500 ppm
I20260927 07:09:45.177047 13074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:45 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492985, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:45.180409 13074 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:09:45.181937 13074 webserver.cc:460] Webserver started at http://127.30.16.67:46055/ using document root <none> and password file <none>
I20260927 07:09:45.182718 13074 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:45.182869 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:45.188275 13074 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 07:09:45.191123 13089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:45.192591 13074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 07:09:45.192776 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "81bcf4f44471488fac08b5a1829dce9e"
format_stamp: "Formatted at 2026-09-27 07:09:42 on dist-test-slave-8vqf"
server_key: "592e0dc22363c3336f4ad7eeb472fe00"
server_key_iv: "694b9b9c66936c496e9b3adb531a6b1e"
server_key_version: "encryptionkey@0"
I20260927 07:09:45.193784 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:45.214653 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:45.219323 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:45.219617 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:45.220687 13074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:45.222121 13074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:45.222254 13074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:45.222396 13074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:45.222481 13074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:45.258081 13074 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:42013
I20260927 07:09:45.258147 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:42013 every 8 connection(s)
I20260927 07:09:45.259626 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:09:45.263833 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13074
Sep 27 07:09:45 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492985, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:09:45.281064 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.264796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.30.16.126:43325)
0927 07:09:45.265791 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.265798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.265819 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:09:45.265902 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.265907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.266006 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.266132 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:45.267623 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.268844 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.270657 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.271055 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.271151 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.274825 (+ 3674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:45.274829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:45.274832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:45.274861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:45.278095 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.279025 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.279030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.279035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.279230 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.279664 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.279668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.279672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.280071 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:45.280337 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.280622 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.280790 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":762,"threads_started":1}
I20260927 07:09:45.283634 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.262097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:57709 (local address 127.30.16.126:43325)
0927 07:09:45.262278 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.262282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.263768 (+ 1486us) server_negotiation.cc:407] Connection header received
0927 07:09:45.266719 (+ 2951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.266724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.266802 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.266920 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:45.268940 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.269657 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.272089 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.272503 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.272597 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.276890 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:45.276896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:45.276900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:45.276949 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:45.280398 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.281331 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.281339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.281343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.281473 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.281915 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.281920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.281923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.282179 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:45.282319 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.283087 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.283293 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:09:45.284631 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.262610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:57709)
0927 07:09:45.263520 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.263564 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.266399 (+ 2835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.267186 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.267194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.267916 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:45.268731 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.268756 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.269826 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.269832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.271923 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.271937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.272757 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.272767 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.273132 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.273924 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:45.273970 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:45.276668 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:45.280542 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:45.280552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:45.280571 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:45.281213 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:45.281591 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:45.281596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:45.281600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:45.281798 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:45.282303 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:45.282311 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:45.282734 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.283001 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.283550 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":371,"threads_started":1}
I20260927 07:09:45.285899 13202 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43325
I20260927 07:09:45.286228 13202 heartbeater.cc:464] Registering TS with master...
I20260927 07:09:45.287050 13202 heartbeater.cc:511] Master 127.30.16.126:43325 requested a full tablet report, sending...
I20260927 07:09:45.288404 12753 ts_manager.cc:194] Registered new tserver with Master: 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:45.289237 12753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:57709
I20260927 07:09:45.296375 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:09:45.303386 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.297348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.30.16.126:43325)
0927 07:09:45.297532 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.297537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.297949 (+ 412us) server_negotiation.cc:407] Connection header received
0927 07:09:45.298212 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.298216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.298395 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.298578 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:09:45.299023 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.299759 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.301240 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.301661 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.301749 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.302241 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:09:45.302816 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:09:45.302924 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.303108 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.303195 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:09:45.304505 12753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:09:45.312850 12753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:09:45.338385 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.326125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54608)
0927 07:09:45.328108 (+ 1983us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.328163 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.328435 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.329540 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.329552 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.329595 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:45.330093 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.330109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.332519 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.332527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.335583 (+ 3056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.335604 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.337438 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.337444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.337632 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.337686 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.337823 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.338029 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1689,"thread_start_us":2106,"threads_started":1}
I20260927 07:09:45.338411 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.326285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.30.16.65:36573)
0927 07:09:45.327368 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.327379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.328269 (+ 890us) server_negotiation.cc:407] Connection header received
0927 07:09:45.328726 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.328736 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.329010 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.329206 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:45.330372 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.332270 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.335784 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.337231 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.337371 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.337622 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.337899 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.338062 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":699,"threads_started":1}
I20260927 07:09:45.339792 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.326199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:42013 (local address 127.0.0.1:33026)
0927 07:09:45.328108 (+ 1909us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.328383 (+ 275us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.328481 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.330517 (+ 2036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.330525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.330582 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:45.331061 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.331076 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.333983 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.333995 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.336464 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.336475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.338954 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.338961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.339165 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.339199 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.339274 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.339391 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1713,"mutex_wait_us":1,"thread_start_us":2431,"threads_started":1}
I20260927 07:09:45.340358 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.326971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.30.16.67:42013)
0927 07:09:45.329504 (+ 2533us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.329513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.329548 (+ 35us) server_negotiation.cc:407] Connection header received
0927 07:09:45.329641 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.329647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.329914 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.330286 (+ 372us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:45.331666 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.333760 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.336621 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.338213 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.338332 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.338626 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.339857 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.340027 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2331,"thread_start_us":2169,"threads_started":1}
I20260927 07:09:45.340724 12874 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 (DEFAULT_TABLE table=test-table [id=46131d5f8bd6493e8c984d8cbfdc870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:45.342187 12874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3a13acb52f4f9b8994d86ec051f364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:45.343968 13137 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 (DEFAULT_TABLE table=test-table [id=46131d5f8bd6493e8c984d8cbfdc870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:45.344666 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.329128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42521 (local address 127.0.0.1:56198)
0927 07:09:45.335888 (+ 6760us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.335906 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.336032 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.336785 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.336790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.336816 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:45.337182 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.337193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.339758 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.339766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.341602 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.341620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.343907 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.343913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.344090 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.344128 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.344217 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.344356 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6633,"thread_start_us":6505,"threads_started":1}
I20260927 07:09:45.345358 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.329606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.30.16.66:42521)
0927 07:09:45.336185 (+ 6579us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.336194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.336224 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:09:45.336326 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.336334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.336575 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.336789 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:45.337363 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.339582 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.341788 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.343189 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.343347 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.343552 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.344898 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.345080 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6415,"thread_start_us":6511,"threads_started":1}
I20260927 07:09:45.345440 13137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3a13acb52f4f9b8994d86ec051f364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:45.348966 13006 tablet_service.cc:1467] Processing CreateTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 (DEFAULT_TABLE table=test-table [id=46131d5f8bd6493e8c984d8cbfdc870a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:09:45.350358 13006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e3a13acb52f4f9b8994d86ec051f364. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:09:45.351760 13218 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Bootstrap starting.
I20260927 07:09:45.352094 12939 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:45.355579 13218 tablet_bootstrap.cc:654] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:45.357163 13218 log.cc:824] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:45.360903 13218 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: No bootstrap required, opened a new log
I20260927 07:09:45.361186 13218 ts_tablet_manager.cc:1397] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 07:09:45.363108 13220 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Bootstrap starting.
I20260927 07:09:45.367121 13220 tablet_bootstrap.cc:654] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:45.368400 13220 log.cc:824] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:45.369854 13221 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Bootstrap starting.
I20260927 07:09:45.371927 13220 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: No bootstrap required, opened a new log
I20260927 07:09:45.371702 13218 raft_consensus.cc:348] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.372187 13218 raft_consensus.cc:374] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:45.372219 13220 ts_tablet_manager.cc:1397] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 07:09:45.372306 13218 raft_consensus.cc:729] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c8b5f362343fd98fb614ba9df3ae0, State: Initialized, Role: FOLLOWER
I20260927 07:09:45.373003 13218 consensus_queue.cc:260] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.373883 13221 tablet_bootstrap.cc:654] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Neither blocks nor log segments found. Creating new log.
I20260927 07:09:45.374948 13218 ts_tablet_manager.cc:1428] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260927 07:09:45.375286 13221 log.cc:824] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:45.378463 13221 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: No bootstrap required, opened a new log
I20260927 07:09:45.378775 13221 ts_tablet_manager.cc:1397] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 07:09:45.383070 13220 raft_consensus.cc:348] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.383548 13220 raft_consensus.cc:374] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:45.383653 13220 raft_consensus.cc:729] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81bcf4f44471488fac08b5a1829dce9e, State: Initialized, Role: FOLLOWER
I20260927 07:09:45.384302 13220 consensus_queue.cc:260] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.385388 13202 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:45.386129 13220 ts_tablet_manager.cc:1428] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 07:09:45.388356 13221 raft_consensus.cc:348] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.388736 13221 raft_consensus.cc:374] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:09:45.388859 13221 raft_consensus.cc:729] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed03c78e28914c45b50f7ef20fef8ac5, State: Initialized, Role: FOLLOWER
I20260927 07:09:45.389451 13221 consensus_queue.cc:260] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.390493 13071 heartbeater.cc:503] Master 127.30.16.126:43325 was elected leader, sending a full tablet report...
I20260927 07:09:45.391264 13221 ts_tablet_manager.cc:1428] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 07:09:45.393687 13226 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:45.393946 13226 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.395560 13226 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573), 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:45.402650 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.395955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:42655 (local address 127.30.16.65:36573)
0927 07:09:45.396131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.396157 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.397084 (+ 927us) server_negotiation.cc:407] Connection header received
0927 07:09:45.397208 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.397213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.397292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.397405 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:45.398146 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.399067 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.400767 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.401593 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.401743 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.401832 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.402227 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.402362 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 07:09:45.402679 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.396102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43129 (local address 127.30.16.67:42013)
0927 07:09:45.396337 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.396345 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.396835 (+ 490us) server_negotiation.cc:407] Connection header received
0927 07:09:45.397004 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.397010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.397086 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.397171 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:09:45.397853 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.398810 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.400871 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.401630 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.401822 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.401898 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.402259 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.402383 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:09:45.402900 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.395820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:42013 (local address 127.30.16.66:43129)
0927 07:09:45.396717 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.396737 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.396863 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.397222 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.397247 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.397273 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:45.397711 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.397727 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.398928 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.398934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.400720 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.400736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.401883 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.401888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.402092 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.402115 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.402219 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.402375 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":72,"spinlock_wait_cycles":37120,"thread_start_us":518,"threads_started":1}
I20260927 07:09:45.402980 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.395913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.30.16.66:42655)
0927 07:09:45.396888 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.396905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:45.397046 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:45.397413 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:45.397416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:45.397483 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:09:45.397921 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.397934 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.399209 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.399213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.400612 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:45.400626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.401820 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.401827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:45.402038 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.402095 (+ 57us) client_negotiation.cc:769] Sending connection context
0927 07:09:45.402173 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 07:09:45.402341 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":22,"spinlock_wait_cycles":5632,"thread_start_us":662,"threads_started":1}
I20260927 07:09:45.403967 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3a13acb52f4f9b8994d86ec051f364" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bcf4f44471488fac08b5a1829dce9e" is_pre_election: true
I20260927 07:09:45.404100 12894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3a13acb52f4f9b8994d86ec051f364" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" is_pre_election: true
I20260927 07:09:45.404627 13157 raft_consensus.cc:2454] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 0.
I20260927 07:09:45.404627 12894 raft_consensus.cc:2454] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 0.
I20260927 07:09:45.405390 12959 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 81bcf4f44471488fac08b5a1829dce9e, e49c8b5f362343fd98fb614ba9df3ae0, ed03c78e28914c45b50f7ef20fef8ac5; no voters:
I20260927 07:09:45.405864 13226 raft_consensus.cc:2781] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:09:45.406014 13226 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:09:45.406122 13226 raft_consensus.cc:3037] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:45.408715 13226 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.409302 13226 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 election: Requested vote from peers e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573), 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013)
I20260927 07:09:45.409852 12894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3a13acb52f4f9b8994d86ec051f364" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
I20260927 07:09:45.409849 13157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e3a13acb52f4f9b8994d86ec051f364" candidate_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81bcf4f44471488fac08b5a1829dce9e"
I20260927 07:09:45.410072 12894 raft_consensus.cc:3037] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:45.410072 13157 raft_consensus.cc:3037] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:09:45.413425 12894 raft_consensus.cc:2454] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 1.
I20260927 07:09:45.413425 13157 raft_consensus.cc:2454] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03c78e28914c45b50f7ef20fef8ac5 in term 1.
I20260927 07:09:45.413883 12959 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5; no voters:
I20260927 07:09:45.414233 13226 raft_consensus.cc:2781] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:09:45.414779 13226 raft_consensus.cc:686] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 LEADER]: Becoming Leader. State: Replica: ed03c78e28914c45b50f7ef20fef8ac5, State: Running, Role: LEADER
I20260927 07:09:45.415297 13226 consensus_queue.cc:237] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:45.420430 12752 catalog_manager.cc:5495] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } health_report { overall_health: HEALTHY } } }
I20260927 07:09:45.439709 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.434157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.30.16.66:42521)
0927 07:09:45.434359 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.434363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.434418 (+ 55us) server_negotiation.cc:407] Connection header received
0927 07:09:45.434673 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.434677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.434769 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.434894 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:09:45.435258 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.435958 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.437380 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.437824 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.437921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.438443 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:09:45.439141 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:09:45.439255 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.439395 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.439470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:09:45.448674 13157 raft_consensus.cc:1261] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Refusing update from remote peer ed03c78e28914c45b50f7ef20fef8ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:45.448679 12894 raft_consensus.cc:1261] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Refusing update from remote peer ed03c78e28914c45b50f7ef20fef8ac5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:09:45.449712 13229 consensus_queue.cc:1035] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:45.450129 13226 consensus_queue.cc:1035] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:09:45.462227 13231 mvcc.cc:205] Tried to move back new op lower bound from 7333859268386295808 to 7333859268265152512. Current Snapshot: MvccSnapshot[applied={T|T < 7333859268386295808}]
I20260927 07:09:45.477610 12752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:45.477941 12752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:09:45.481853 12752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Sending DeleteTablet for 3 replicas of tablet 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:45.483395 13137 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:45 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:33026
I20260927 07:09:45.483366 13006 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:45 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:56198
I20260927 07:09:45.483371 12874 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e3a13acb52f4f9b8994d86ec051f364 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:09:45 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:54608
I20260927 07:09:45.483716 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12790
I20260927 07:09:45.486750 13243 tablet_replica.cc:330] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: stopping tablet replica
I20260927 07:09:45.487347 13243 raft_consensus.cc:2229] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:45.489473 13243 raft_consensus.cc:2258] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:45.497711 13242 tablet_replica.cc:330] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: stopping tablet replica
I20260927 07:09:45.497825 13241 tablet_replica.cc:330] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: stopping tablet replica
I20260927 07:09:45.498368 13242 raft_consensus.cc:2229] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:09:45.498342 13241 raft_consensus.cc:2229] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:09:45.499032 13241 raft_consensus.cc:2258] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:45.499091 13242 raft_consensus.cc:2258] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:09:45.500172 13243 ts_tablet_manager.cc:1905] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:45.501616 13242 ts_tablet_manager.cc:1905] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:45.503471 13241 ts_tablet_manager.cc:1905] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:09:45.504479 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.485523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.30.16.65:36573)
0927 07:09:45.485706 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.485713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.485734 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:09:45.485809 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.485815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.485922 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.486058 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:45.487964 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.489142 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.491040 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.491670 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.491806 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.495350 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:45.495367 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:45.495387 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:45.495442 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:45.499502 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.501017 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.501026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.501030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.501206 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.501680 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.501686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.501689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.502176 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:45.502477 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.503857 (+ 1380us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.504106 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 07:09:45.505215 12934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:45.509253 13242 ts_tablet_manager.cc:1918] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:45.509439 13242 log.cc:1196] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:45.510039 13242 ts_tablet_manager.cc:1939] T 1e3a13acb52f4f9b8994d86ec051f364 P ed03c78e28914c45b50f7ef20fef8ac5: Deleting consensus metadata
I20260927 07:09:45.511921 13241 ts_tablet_manager.cc:1918] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:09:45.512084 12737 catalog_manager.cc:4841] TS ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): tablet 1e3a13acb52f4f9b8994d86ec051f364 (table test-table [id=46131d5f8bd6493e8c984d8cbfdc870a]) successfully deleted
I20260927 07:09:45.512665 13241 log.cc:1196] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:45.513607 13241 ts_tablet_manager.cc:1939] T 1e3a13acb52f4f9b8994d86ec051f364 P 81bcf4f44471488fac08b5a1829dce9e: Deleting consensus metadata
I20260927 07:09:45.515564 12737 catalog_manager.cc:4841] TS 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): tablet 1e3a13acb52f4f9b8994d86ec051f364 (table test-table [id=46131d5f8bd6493e8c984d8cbfdc870a]) successfully deleted
I20260927 07:09:45.608666 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12790
I20260927 07:09:45.627472 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12943
W20260927 07:09:45.627429 12740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.16.65:36573 (error 108)
I20260927 07:09:45.629424 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.628955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54624)
0927 07:09:45.629146 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.629265 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 07:09:45.629752 12740 catalog_manager.cc:4584] TS e49c8b5f362343fd98fb614ba9df3ae0 (127.30.16.65:36573): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e3a13acb52f4f9b8994d86ec051f364: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
I20260927 07:09:45.642906 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.628652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.30.16.66:42521)
0927 07:09:45.628821 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.628825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.628842 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:09:45.628897 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.628901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.629018 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.629127 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:45.630956 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.631744 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.633298 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.633763 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.633898 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.636711 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:45.636724 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:45.636739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:45.636776 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:45.639880 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.640625 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.640632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.640644 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.640786 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.641174 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.641183 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.641191 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.641620 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:45.641981 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.642279 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.642505 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:09:45.643638 13066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:45.686390 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.685781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54630)
0927 07:09:45.686075 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.686201 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 07:09:45.687613 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.687222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54640)
0927 07:09:45.687397 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.687469 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 07:09:45.748422 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.747793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54642)
0927 07:09:45.748082 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.748205 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 07:09:45.749743 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.749286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54650)
0927 07:09:45.749466 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.749569 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 07:09:45.792116 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12943
I20260927 07:09:45.811309 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13074
I20260927 07:09:45.825006 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.812180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.30.16.67:42013)
0927 07:09:45.812341 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:45.812345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:45.812364 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:09:45.812579 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:45.812583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:45.812687 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:45.812821 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:45.814279 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.815089 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.816374 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:45.816790 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:45.816887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:45.819462 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:45.819476 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:45.819492 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:45.819522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:45.822312 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.823021 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.823028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.823030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.823210 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:45.823541 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:45.823544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:45.823546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:45.823877 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:45.824096 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:45.824449 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 07:09:45.824619 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 07:09:45.825662 13197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:45.830420 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.829974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54658)
0927 07:09:45.830163 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.830253 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 07:09:45.831568 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:45.831181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54664)
0927 07:09:45.831393 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:45.831460 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 07:09:45.916002 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13074
I20260927 07:09:45.933290 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 12720
I20260927 07:09:45.934417 12783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:09:46.052379 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 12720
I20260927 07:09:46.052459 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:46.051743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36573 (local address 127.0.0.1:54666)
0927 07:09:46.052014 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:46.052158 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:36573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260927 07:09:46.075344 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--webserver_interface=127.30.16.126
--webserver_port=40917
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:46.416339 13249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:46.416724 13249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:46.416908 13249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:46.424337 13249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:09:46.424515 13249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:46.424625 13249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:46.424726 13249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:09:46.424811 13249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:09:46.435209 13249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=40917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:46.436678 13249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:46.438064 13249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:46.450966 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:46.451066 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:46.451093 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:46.451722 13249 server_base.cc:1034] running on GCE node
I20260927 07:09:46.452536 13249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:46.453945 13249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:46.455191 13249 hybrid_clock.cc:648] HybridClock initialized: now 1790492986455111 us; error 59 us; skew 500 ppm
I20260927 07:09:46.455617 13249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:46 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:46.459199 13249 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:09:46.460780 13249 webserver.cc:460] Webserver started at http://127.30.16.126:40917/ using document root <none> and password file <none>
I20260927 07:09:46.461458 13249 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:46.461591 13249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:46.467326 13249 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 07:09:46.470302 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:46.471964 13249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:09:46.472234 13249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "fb484acc770d49648841dd59f09b85a5"
format_stamp: "Formatted at 2026-09-27 07:09:40 on dist-test-slave-8vqf"
server_key: "05d0b68547e5ea58f5ef1ea6cdd524e9"
server_key_iv: "be616b14d636573345453adc6c44988c"
server_key_version: "encryptionkey@0"
I20260927 07:09:46.473308 13249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:46.508080 13249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:46.512503 13249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:46.512807 13249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:46.532295 13249 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43325
I20260927 07:09:46.532315 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43325 every 8 connection(s)
I20260927 07:09:46.534133 13249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:09:46.539835 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13249
I20260927 07:09:46.540822 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:36573
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=45319
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:46.542951 13316 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:09:46.546458 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap starting.
I20260927 07:09:46.577216 13316 log.cc:824] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:46.598646 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:09:46.599393 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5: Bootstrap complete.
I20260927 07:09:46.611258 13316 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:46.612210 13316 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Initialized, Role: FOLLOWER
I20260927 07:09:46.612845 13316 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:46.613078 13316 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:09:46.613189 13316 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:09:46.613373 13316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:09:46.617324 13316 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:46.617847 13316 leader_election.cc:302] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb484acc770d49648841dd59f09b85a5; no voters:
I20260927 07:09:46.618486 13316 leader_election.cc:288] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:09:46.618698 13320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:09:46.619764 13320 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [term 3 LEADER]: Becoming Leader. State: Replica: fb484acc770d49648841dd59f09b85a5, State: Running, Role: LEADER
I20260927 07:09:46.620360 13320 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } }
I20260927 07:09:46.620743 13316 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:09:46.622859 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:46.623225 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:46.623623 13322 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb484acc770d49648841dd59f09b85a5. Latest consensus state: current_term: 3 leader_uuid: "fb484acc770d49648841dd59f09b85a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb484acc770d49648841dd59f09b85a5" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43325 } } }
I20260927 07:09:46.623864 13322 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb484acc770d49648841dd59f09b85a5 [sys.catalog]: This master's current role is: LEADER
I20260927 07:09:46.626363 13325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:09:46.636529 13325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:09:46.638665 13325 catalog_manager.cc:1259] Loaded cluster ID: 99ffd0400b5e41ebb3df24f52c47d334
I20260927 07:09:46.638839 13325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 07:09:46.641691 13325 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f04138149ad google::LogMessage::Fail() at ??:0
@ 0x7f041381788a google::LogMessage::SendToLog() at ??:0
@ 0x7f04138144de google::LogMessage::Flush() at ??:0
@ 0x7f0413815a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0429247512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0429244976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f04292cc610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f04292cc439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0416460345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f041646391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0416463799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f041643eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f04197186db start_thread at ??:0
@ 0x7f04126ae61f clone at ??:0
W20260927 07:09:46.940935 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:46.941282 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:46.941449 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:46.949550 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:46.949718 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:46.949905 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:09:46.960989 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:36573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=45319
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:46.962616 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:46.964150 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:46.978820 13358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:46.978955 13357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:46.979241 13360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:46.980036 13318 server_base.cc:1034] running on GCE node
I20260927 07:09:46.980763 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:46.981834 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:46.983206 13318 hybrid_clock.cc:648] HybridClock initialized: now 1790492986983117 us; error 75 us; skew 500 ppm
I20260927 07:09:46.983551 13318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:46 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:46.987923 13318 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:09:46.989436 13318 webserver.cc:460] Webserver started at http://127.30.16.65:45319/ using document root <none> and password file <none>
I20260927 07:09:46.990198 13318 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:46.990350 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:46.996421 13318 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 07:09:46.999625 13367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.001237 13318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 07:09:47.001435 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e49c8b5f362343fd98fb614ba9df3ae0"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "a2d5f411db93126fd124f7006244e79c"
server_key_iv: "aa2faf5742451503866ec1a2377d3b05"
server_key_version: "encryptionkey@0"
I20260927 07:09:47.002482 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:47.018797 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:47.023063 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:47.023357 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:47.024406 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:47.026350 13374 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 07:09:47.032302 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 07:09:47.032488 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260927 07:09:47.032642 13318 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 07:09:47.036175 13318 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 07:09:47.036291 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260927 07:09:47.036679 13374 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Bootstrap starting.
I20260927 07:09:47.045703 13374 log.cc:824] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:09:47.066344 13374 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:09:47.067297 13374 tablet_bootstrap.cc:492] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Bootstrap complete.
I20260927 07:09:47.068166 13374 ts_tablet_manager.cc:1397] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.008s
I20260927 07:09:47.076181 13318 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36573
I20260927 07:09:47.076225 13481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36573 every 8 connection(s)
I20260927 07:09:47.078064 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:09:47.079059 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13318
I20260927 07:09:47.079855 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:42521
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=43145
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:09:47.082358 13374 raft_consensus.cc:348] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:47.084488 13374 raft_consensus.cc:729] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e49c8b5f362343fd98fb614ba9df3ae0, State: Initialized, Role: FOLLOWER
I20260927 07:09:47.085626 13374 consensus_queue.cc:260] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:47.094895 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.081600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:60493)
0927 07:09:47.093396 (+ 11796us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.093791 (+ 395us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11377,"threads_started":1}
W20260927 07:09:47.095824 13482 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43325 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43325: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
I20260927 07:09:47.096721 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.096384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:45543)
0927 07:09:47.096542 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.096616 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 07:09:47.097553 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.097219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:38297)
0927 07:09:47.097358 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.097429 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 07:09:47.097774 13482 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:09:47.102377 13374 ts_tablet_manager.cc:1428] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0: Time spent starting tablet: real 0.034s user 0.016s sys 0.007s
I20260927 07:09:47.103358 13484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.103028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:42659)
0927 07:09:47.103183 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.103251 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:09:47.440186 13486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:47.440550 13486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:47.440713 13486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:47.448489 13486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:47.448660 13486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:47.448848 13486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:09:47.459152 13486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:42521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=43145
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:47.460577 13486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:47.461856 13486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:47.476830 13491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:47.476764 13494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:47.476796 13492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:47.477267 13486 server_base.cc:1034] running on GCE node
I20260927 07:09:47.478042 13486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:47.479132 13486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:47.480400 13486 hybrid_clock.cc:648] HybridClock initialized: now 1790492987480333 us; error 67 us; skew 500 ppm
I20260927 07:09:47.480710 13486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:47 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492987, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:47.484129 13486 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:09:47.485628 13486 webserver.cc:460] Webserver started at http://127.30.16.66:43145/ using document root <none> and password file <none>
I20260927 07:09:47.486310 13486 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:47.486419 13486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:47.492245 13486 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 07:09:47.495260 13501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.496726 13486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:09:47.496865 13486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "ed03c78e28914c45b50f7ef20fef8ac5"
format_stamp: "Formatted at 2026-09-27 07:09:41 on dist-test-slave-8vqf"
server_key: "e6e9a2fffc73c148f7728c5d94db3e04"
server_key_iv: "41d92eeb59873a3f1fef001b198d8351"
server_key_version: "encryptionkey@0"
I20260927 07:09:47.497854 13486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:47.519876 13486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:47.526489 13486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:47.526803 13486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:47.528016 13486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:47.529686 13486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:47.529776 13486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.529865 13486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:47.529922 13486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.564662 13486 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:42521
I20260927 07:09:47.564718 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:42521 every 8 connection(s)
I20260927 07:09:47.566152 13486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:09:47.571362 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13486
I20260927 07:09:47.571576 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.569216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:33623)
0927 07:09:47.570076 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.570302 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260927 07:09:47.572160 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:42013
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:43325
--webserver_port=46055
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:09:47.572522 13614 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43325 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43325: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
I20260927 07:09:47.573621 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.573151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:60453)
0927 07:09:47.573355 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.573460 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 07:09:47.575150 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:47.574700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:56591)
0927 07:09:47.574875 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:47.574990 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:09:47.575402 13614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:09:47.907123 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:09:47.907421 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:09:47.907600 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:09:47.915156 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:09:47.915271 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:09:47.915447 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:09:47.925786 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:42013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=46055
--tserver_master_addrs=127.30.16.126:43325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:09:47.927223 13617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:09:47.928584 13617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:09:47.942713 13622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:47.942775 13623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:09:47.942806 13625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:09:47.943821 13617 server_base.cc:1034] running on GCE node
I20260927 07:09:47.944450 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:09:47.945416 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:09:47.946660 13617 hybrid_clock.cc:648] HybridClock initialized: now 1790492987946605 us; error 39 us; skew 500 ppm
I20260927 07:09:47.947098 13617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:09:47 dist-test-slave-8vqf krb5kdc[12167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492987, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:09:47.950742 13617 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:09:47.952404 13617 webserver.cc:460] Webserver started at http://127.30.16.67:46055/ using document root <none> and password file <none>
I20260927 07:09:47.953158 13617 fs_manager.cc:359] Metadata directory not provided
I20260927 07:09:47.953301 13617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:09:47.959435 13617 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:09:47.962630 13632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.964187 13617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 07:09:47.964377 13617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "81bcf4f44471488fac08b5a1829dce9e"
format_stamp: "Formatted at 2026-09-27 07:09:42 on dist-test-slave-8vqf"
server_key: "592e0dc22363c3336f4ad7eeb472fe00"
server_key_iv: "694b9b9c66936c496e9b3adb531a6b1e"
server_key_version: "encryptionkey@0"
I20260927 07:09:47.965471 13617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:09:47.988785 13617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:09:47.994884 13617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:09:47.995234 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:09:47.996336 13617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:09:47.997850 13617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:09:47.997943 13617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:47.998104 13617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:09:47.998174 13617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:09:48.032618 13617 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:42013
I20260927 07:09:48.032733 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:42013 every 8 connection(s)
I20260927 07:09:48.034127 13617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:09:48.036561 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13617
I20260927 07:09:48.039372 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.037231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:35815)
0927 07:09:48.038126 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.038401 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":396,"threads_started":1}
W20260927 07:09:48.040035 13745 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43325 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43325: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
I20260927 07:09:48.040849 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.040512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:50737)
0927 07:09:48.040688 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.040749 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 07:09:48.041736 13747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.041382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:55385)
0927 07:09:48.041548 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.041629 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 07:09:48.041973 13745 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:09:48.105234 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.104259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:57817)
0927 07:09:48.104782 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.104911 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":252,"threads_started":1}
I20260927 07:09:48.577059 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.576016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:33813)
0927 07:09:48.576615 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.576748 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":335,"threads_started":1}
I20260927 07:09:48.656350 13751 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:48.656723 13751 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:48.658396 13751 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013), ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
Sep 27 07:09:48 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
Sep 27 07:09:48 dist-test-slave-8vqf krb5kdc[12167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790492986, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:09:48.681180 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.659318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:42521 (local address 127.30.16.65:57201)
0927 07:09:48.660123 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.660143 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:48.660307 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:48.662797 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:48.662802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:48.663631 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:48.664338 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:48.664377 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.665874 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.665878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:48.667419 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:48.667431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.668890 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.668895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:48.669021 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:48.669758 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:48.669800 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:48.672479 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:48.677145 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:48.677152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:48.677172 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:48.677632 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:48.678048 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:48.678053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:48.678057 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:48.678319 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:48.679442 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:48.679448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:48.679761 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 07:09:48.680096 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 07:09:48.680797 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":422,"spinlock_wait_cycles":28800,"thread_start_us":489,"threads_started":1}
I20260927 07:09:48.681205 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.658615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:42013 (local address 127.30.16.65:51611)
0927 07:09:48.659184 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:48.659236 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:09:48.659934 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:09:48.662717 (+ 2783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:09:48.662731 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:09:48.663631 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:09:48.664338 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:48.664358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.665876 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.665882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:48.667431 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:09:48.667462 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.668601 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.668608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:09:48.668901 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:48.669803 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:09:48.669823 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:09:48.672746 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:09:48.677754 (+ 5008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:48.677757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:48.677759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:48.678173 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:48.678708 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:09:48.678713 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:09:48.678716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:09:48.678870 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:09:48.679792 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:09:48.679801 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:09:48.679941 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 07:09:48.680096 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 07:09:48.680681 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":61,"thread_start_us":239,"threads_started":1}
I20260927 07:09:48.681236 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.659908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:57201 (local address 127.30.16.66:42521)
0927 07:09:48.660574 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:48.660586 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:48.660623 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:09:48.662136 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:48.662152 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:48.662478 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:48.662837 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:48.664538 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.665683 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.667915 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.668653 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.668869 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:48.672771 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:48.672778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:48.672793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:48.672860 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:48.676986 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:48.677750 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:48.677756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:48.677762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:48.677910 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:48.678449 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:48.678455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:48.678461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:48.679298 (+ 837us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:48.679496 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:48.680389 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 07:09:48.680832 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":353,"threads_started":1}
I20260927 07:09:48.681901 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:48.658949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:51611 (local address 127.30.16.67:42013)
0927 07:09:48.659504 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 07:09:48.659520 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:09:48.659564 (+ 44us) server_negotiation.cc:407] Connection header received
0927 07:09:48.661827 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:09:48.661836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:09:48.662226 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:09:48.662602 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:09:48.664539 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.665707 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.667627 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:09:48.668388 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:09:48.668589 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:09:48.673057 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:09:48.673069 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:09:48.673089 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:09:48.673136 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:09:48.676986 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:48.678329 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:48.678338 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:48.678346 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:48.678584 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:09:48.679022 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:09:48.679027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:09:48.679030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:09:48.679672 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:09:48.679816 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 07:09:48.680389 (+ 573us) server_negotiation.cc:299] Negotiation successful
0927 07:09:48.680842 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
W20260927 07:09:48.684388 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
W20260927 07:09:48.684672 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:48.684773 13368 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0; no voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5
I20260927 07:09:48.685115 13751 raft_consensus.cc:2726] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:09:49.043582 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:49.042537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:42231)
0927 07:09:49.043131 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:49.043309 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":241,"threads_started":1}
I20260927 07:09:49.106748 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:49.106236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:38311)
0927 07:09:49.106422 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:49.106523 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 07:09:49.578967 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:49.578133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:57173)
0927 07:09:49.578702 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:49.578794 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":272,"threads_started":1}
I20260927 07:09:50.045223 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:50.044524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:42541)
0927 07:09:50.044957 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:50.045037 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":240,"threads_started":1}
I20260927 07:09:50.108428 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:50.107704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:51565)
0927 07:09:50.108152 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:50.108250 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":221,"threads_started":1}
I20260927 07:09:50.580915 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:50.580084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:44147)
0927 07:09:50.580623 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:50.580726 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":262,"threads_started":1}
I20260927 07:09:50.669476 13761 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:50.669708 13761 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:50.670357 13761 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013), ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
W20260927 07:09:50.671464 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
W20260927 07:09:50.671748 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:50.671831 13368 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0; no voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5
I20260927 07:09:50.672166 13761 raft_consensus.cc:2726] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:09:51.047109 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:51.046266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:51615)
0927 07:09:51.046862 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:51.046947 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":334,"threads_started":1}
I20260927 07:09:51.110241 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:51.109335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:33831)
0927 07:09:51.109891 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:51.110025 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":230,"threads_started":1}
I20260927 07:09:51.582707 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:51.581811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:45829)
0927 07:09:51.582344 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:51.582428 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":314,"threads_started":1}
I20260927 07:09:52.048816 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:52.048034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:54957)
0927 07:09:52.048544 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:52.048632 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":266,"threads_started":1}
I20260927 07:09:52.112134 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:52.111383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:52283)
0927 07:09:52.111828 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:52.111913 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":247,"threads_started":1}
I20260927 07:09:52.584470 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:52.583747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:35595)
0927 07:09:52.584215 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:52.584303 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":232,"threads_started":1}
I20260927 07:09:53.050601 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:53.049776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:43777)
0927 07:09:53.050288 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:53.050401 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
I20260927 07:09:53.113835 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:53.113015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:53691)
0927 07:09:53.113554 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:53.113651 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":311,"threads_started":1}
I20260927 07:09:53.586468 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:53.585567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:60865)
0927 07:09:53.586146 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:53.586243 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
I20260927 07:09:53.817826 13771 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:53.818118 13771 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:53.818859 13771 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013), ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
W20260927 07:09:53.819875 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
W20260927 07:09:53.820083 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:53.820201 13368 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0; no voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5
I20260927 07:09:53.820447 13771 raft_consensus.cc:2726] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:09:54.052366 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:54.051582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:58771)
0927 07:09:54.052092 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:54.052178 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260927 07:09:54.115970 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:54.114982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:41719)
0927 07:09:54.115569 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:54.115680 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":283,"threads_started":1}
I20260927 07:09:54.588750 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:54.587696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:44777)
0927 07:09:54.588359 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:54.588478 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":276,"threads_started":1}
I20260927 07:09:55.054214 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:55.053338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:50431)
0927 07:09:55.053892 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:55.053989 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260927 07:09:55.117944 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:55.116974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:47825)
0927 07:09:55.117549 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:55.117651 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":310,"threads_started":1}
I20260927 07:09:55.590737 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:55.589797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:57155)
0927 07:09:55.590362 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:55.590455 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":330,"threads_started":1}
I20260927 07:09:55.967224 13778 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:09:55.967468 13778 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:09:55.968246 13778 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013), ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
W20260927 07:09:55.969130 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
W20260927 07:09:55.969412 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:09:55.969498 13368 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0; no voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5
I20260927 07:09:55.969821 13778 raft_consensus.cc:2726] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:09:56.056058 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:56.055298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:48011)
0927 07:09:56.055775 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:56.055868 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":254,"threads_started":1}
I20260927 07:09:56.119861 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:56.119025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:47875)
0927 07:09:56.119557 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:56.119682 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":263,"threads_started":1}
I20260927 07:09:56.592510 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:56.591712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:60863)
0927 07:09:56.592253 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:56.592351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260927 07:09:57.057904 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:57.057179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:34527)
0927 07:09:57.057651 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:57.057749 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":223,"threads_started":1}
I20260927 07:09:57.121713 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:57.120958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:40127)
0927 07:09:57.121422 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:57.121512 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
I20260927 07:09:57.594359 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:57.593621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:34901)
0927 07:09:57.594080 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:57.594177 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":242,"threads_started":1}
I20260927 07:09:58.059870 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:58.059156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:45461)
0927 07:09:58.059600 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:58.059690 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":220,"threads_started":1}
I20260927 07:09:58.123564 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:58.122859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:55357)
0927 07:09:58.123297 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:58.123391 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260927 07:09:58.596305 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:58.595455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:53681)
0927 07:09:58.596027 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:58.596134 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
I20260927 07:09:59.061620 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:59.060807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:41965)
0927 07:09:59.061336 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:59.061430 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260927 07:09:59.125407 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:59.124596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:52503)
0927 07:09:59.125094 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:59.125206 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":265,"threads_started":1}
I20260927 07:09:59.598066 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:09:59.597334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:55085)
0927 07:09:59.597799 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 07:09:59.597891 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":238,"threads_started":1}
I20260927 07:10:00.063868 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:00.062875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:59195)
0927 07:10:00.063491 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:00.063603 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":319,"threads_started":1}
I20260927 07:10:00.127327 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:00.126464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:38247)
0927 07:10:00.126996 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:00.127135 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":299,"threads_started":1}
I20260927 07:10:00.599880 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:00.599125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:50061)
0927 07:10:00.599615 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:00.599710 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
I20260927 07:10:00.685736 13794 raft_consensus.cc:482] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:10:00.685930 13794 raft_consensus.cc:504] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e49c8b5f362343fd98fb614ba9df3ae0" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36573 } } peers { permanent_uuid: "81bcf4f44471488fac08b5a1829dce9e" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 42013 } } peers { permanent_uuid: "ed03c78e28914c45b50f7ef20fef8ac5" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 42521 } }
I20260927 07:10:00.686657 13794 leader_election.cc:288] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013), ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521)
W20260927 07:10:00.687817 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 81bcf4f44471488fac08b5a1829dce9e (127.30.16.67:42013): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
W20260927 07:10:00.688050 13368 leader_election.cc:341] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ed03c78e28914c45b50f7ef20fef8ac5 (127.30.16.66:42521): Not found: Tablet not found: 1e3a13acb52f4f9b8994d86ec051f364
I20260927 07:10:00.688175 13368 leader_election.cc:302] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e49c8b5f362343fd98fb614ba9df3ae0; no voters: 81bcf4f44471488fac08b5a1829dce9e, ed03c78e28914c45b50f7ef20fef8ac5
I20260927 07:10:00.688452 13794 raft_consensus.cc:2726] T 1e3a13acb52f4f9b8994d86ec051f364 P e49c8b5f362343fd98fb614ba9df3ae0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 07:10:01.065776 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:01.065025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:41303)
0927 07:10:01.065486 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:01.065608 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260927 07:10:01.129282 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:01.128369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:46223)
0927 07:10:01.128888 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:01.128982 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":254,"threads_started":1}
I20260927 07:10:01.601831 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:01.600871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:48061)
0927 07:10:01.601391 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:01.601491 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":292,"threads_started":1}
I20260927 07:10:02.067844 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:02.066904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:53067)
0927 07:10:02.067482 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:02.067591 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":278,"threads_started":1}
I20260927 07:10:02.131558 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:02.130819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.65:49817)
0927 07:10:02.131279 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:02.131377 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260927 07:10:02.604211 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:02.603194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.66:41513)
0927 07:10:02.603856 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:02.604019 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":343,"threads_started":1}
I20260927 07:10:03.046623 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13318
I20260927 07:10:03.070124 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:03.069202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43325 (local address 127.30.16.67:49563)
0927 07:10:03.069783 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:03.069921 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":305,"threads_started":1}
I20260927 07:10:03.072917 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:03.047878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.30.16.65:36573)
0927 07:10:03.048401 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:03.048409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:03.048689 (+ 280us) server_negotiation.cc:407] Connection header received
0927 07:10:03.048903 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:03.048910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:03.049140 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:03.049351 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:03.051370 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.052431 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.053950 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.054631 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.054752 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:03.066035 (+ 11283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:03.066054 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:03.066071 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:03.066126 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:03.069597 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.070647 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.070656 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.070658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.070796 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.071238 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.071242 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.071244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.071609 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:03.071915 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:03.072257 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 07:10:03.072560 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":229,"threads_started":1}
I20260927 07:10:03.074255 13477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:03.165956 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13318
I20260927 07:10:03.184799 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13486
I20260927 07:10:03.198319 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:03.185560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.30.16.66:42521)
0927 07:10:03.186038 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:03.186042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:03.186060 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:10:03.186197 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:03.186200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:03.186261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:03.186354 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:03.187776 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.188535 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.189803 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.190252 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.190350 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:03.193175 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:03.193180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:03.193183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:03.193218 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:03.195971 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.196616 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.196619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.196621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.196746 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.197090 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.197093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.197095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.197356 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:03.197557 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:03.197797 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 07:10:03.197969 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":262,"threads_started":1}
I20260927 07:10:03.199365 13609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:03.287984 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13486
I20260927 07:10:03.306329 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13617
I20260927 07:10:03.321617 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:03.307412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.30.16.67:42013)
0927 07:10:03.308152 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:03.308157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:03.308176 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:10:03.308233 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:03.308236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:03.308307 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:03.308405 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:03.309904 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.310927 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.312213 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:03.312674 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:03.312820 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:03.315824 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:03.315830 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:03.315833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:03.315872 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:03.318801 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.319525 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.319532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.319536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.319693 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:03.320095 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:03.320101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:03.320104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:03.320382 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:03.320673 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:03.320968 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 07:10:03.321214 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":405,"threads_started":1}
I20260927 07:10:03.322453 13740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:03.410121 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13617
2026-09-27T07:10:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24475 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:10:03 dist-test-slave-8vqf krb5kdc[13811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:10:03 dist-test-slave-8vqf krb5kdc[13811](info): set up 2 sockets
Sep 27 07:10:03 dist-test-slave-8vqf krb5kdc[13811](info): commencing operation
krb5kdc: starting...
W20260927 07:10:04.527225 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:10:04 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:10:04.783247 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:05.121905 13824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:05.122282 13824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:05.122437 13824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:05.130072 13824 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:10:05.130223 13824 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:10:05.130312 13824 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:10:05.130407 13824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:10:05.130508 13824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:05.130661 13824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:05.130754 13824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:10:05.130837 13824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:10:05.141263 13824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:05.142733 13824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:05.144164 13824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:05.157176 13829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:05.157138 13832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:05.157101 13830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:05.157466 13824 server_base.cc:1034] running on GCE node
I20260927 07:10:05.158782 13824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:05.160326 13824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:05.161559 13824 hybrid_clock.cc:648] HybridClock initialized: now 1790493005161487 us; error 44 us; skew 500 ppm
I20260927 07:10:05.161933 13824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:05 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493005, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:05.165500 13824 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:10:05.167153 13824 webserver.cc:460] Webserver started at http://127.30.16.126:38979/ using document root <none> and password file <none>
I20260927 07:10:05.167937 13824 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:05.168082 13824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:05.168419 13824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:10:05.171736 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "5be94b8ece764074bd4abe5e970e814d"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "3fd7556322451c661933538e4207a431"
server_key_iv: "9580087c1e871de9cfe3356c818ff6ac"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.172581 13824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "5be94b8ece764074bd4abe5e970e814d"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "3fd7556322451c661933538e4207a431"
server_key_iv: "9580087c1e871de9cfe3356c818ff6ac"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.178774 13824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260927 07:10:05.182647 13839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:05.184151 13824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:05.184355 13824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5be94b8ece764074bd4abe5e970e814d"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "3fd7556322451c661933538e4207a431"
server_key_iv: "9580087c1e871de9cfe3356c818ff6ac"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.184612 13824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:05.197297 13824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:05.201895 13824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:05.202239 13824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:05.221853 13824 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43019
I20260927 07:10:05.221877 13890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43019 every 8 connection(s)
I20260927 07:10:05.223464 13824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:10:05.226976 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13824
I20260927 07:10:05.227229 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:10:05.228206 30785 external_mini_cluster.cc:1428] Setting key 15fd7f49086f364c331979a4682d8e1b
I20260927 07:10:05.228294 13891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:05.241622 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap starting.
Sep 27 07:10:05 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:05.245199 13891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:05.246464 13891 log.cc:824] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:05.249380 13891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: No bootstrap required, opened a new log
I20260927 07:10:05.252591 13894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:05.230497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.30.16.126:43019)
0927 07:10:05.231313 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:05.231325 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:05.231367 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:10:05.233471 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:05.233495 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:05.233914 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:05.234381 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:05.235906 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.237195 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.238602 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.239321 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.239540 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:05.244189 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:05.244199 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:05.244215 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:05.244261 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:05.247970 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:05.248792 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:05.248806 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:05.248814 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:05.248994 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:05.249474 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:05.249483 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:05.249489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:05.250047 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:05.250433 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:05.250935 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 07:10:05.251439 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":336,"threads_started":1}
I20260927 07:10:05.258311 13891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:05.258812 13891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:05.258925 13891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Initialized, Role: FOLLOWER
I20260927 07:10:05.259505 13891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:05.259749 13891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:10:05.259910 13891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:10:05.260102 13891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:05.262326 13891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:05.262909 13891 leader_election.cc:302] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5be94b8ece764074bd4abe5e970e814d; no voters:
I20260927 07:10:05.263581 13891 leader_election.cc:288] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:10:05.263767 13896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:10:05.264667 13896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 LEADER]: Becoming Leader. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Running, Role: LEADER
I20260927 07:10:05.265281 13896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:05.265838 13891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:10:05.268610 13898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be94b8ece764074bd4abe5e970e814d. Latest consensus state: current_term: 1 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:05.268581 13897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:05.269150 13898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:05.269155 13897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:05.272208 13902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:10:05.278771 13902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:10:05.290014 13902 catalog_manager.cc:1347] Generated new cluster ID: abf825cee14240ea8b4208f34af1ee70
I20260927 07:10:05.290141 13902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:10:05.325186 13902 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:10:05.326704 13902 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:10:05.338106 13902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Generated new TSK 0
I20260927 07:10:05.339138 13902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:10:05.410835 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:05.759021 13917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:05.759399 13917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:05.759570 13917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:05.767315 13917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:05.767500 13917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:05.767688 13917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:10:05.778177 13917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:05.779632 13917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:05.781011 13917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:05.795100 13922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:05.795205 13923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:05.795118 13925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:05.796314 13917 server_base.cc:1034] running on GCE node
I20260927 07:10:05.797144 13917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:05.798290 13917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:05.799746 13917 hybrid_clock.cc:648] HybridClock initialized: now 1790493005799662 us; error 71 us; skew 500 ppm
I20260927 07:10:05.800091 13917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:05 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493005, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:05.806202 13917 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:10:05.809073 13917 webserver.cc:460] Webserver started at http://127.30.16.65:45339/ using document root <none> and password file <none>
I20260927 07:10:05.810367 13917 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:05.810585 13917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:05.811033 13917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:10:05.816308 13917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "267dfef0c6224ae2af395d4eda085ddd"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "0590c87ef2eb0ef1defad69f0205fd12"
server_key_iv: "8cceb54977e74f45f95c966fcad544bc"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.817767 13917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "267dfef0c6224ae2af395d4eda085ddd"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "0590c87ef2eb0ef1defad69f0205fd12"
server_key_iv: "8cceb54977e74f45f95c966fcad544bc"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.826382 13917 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.010s sys 0.001s
I20260927 07:10:05.831949 13932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:05.833880 13917 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260927 07:10:05.834084 13917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "267dfef0c6224ae2af395d4eda085ddd"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "0590c87ef2eb0ef1defad69f0205fd12"
server_key_iv: "8cceb54977e74f45f95c966fcad544bc"
server_key_version: "encryptionkey@0"
I20260927 07:10:05.834343 13917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:05.854857 13917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:05.859922 13917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:05.860194 13917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:05.861207 13917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:05.862599 13917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:05.862699 13917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:05.862819 13917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:05.862879 13917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:05.899751 13917 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45537
I20260927 07:10:05.899804 14044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45537 every 8 connection(s)
I20260927 07:10:05.901084 13917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:10:05.909766 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 13917
I20260927 07:10:05.910005 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:10:05.910734 30785 external_mini_cluster.cc:1428] Setting key 2fbae254d8c124dbf4d0fcb5282fd738
Sep 27 07:10:05 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493005, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:05.923193 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:05.903744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:33485 (local address 127.30.16.126:43019)
0927 07:10:05.904227 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:05.904232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:05.905095 (+ 863us) server_negotiation.cc:407] Connection header received
0927 07:10:05.907041 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:05.907047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:05.907145 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:05.907264 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:05.908823 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.909778 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.911595 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.912079 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.912186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:05.915793 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:05.915802 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:05.915810 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:05.915854 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:05.919560 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:05.920522 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:05.920529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:05.920533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:05.920667 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:05.921155 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:05.921159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:05.921162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:05.921451 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:05.921633 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:05.922687 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0927 07:10:05.922897 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20260927 07:10:05.924767 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:05.904224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:33485)
0927 07:10:05.904925 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:05.904965 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:05.906762 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:05.907510 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:05.907518 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:05.908047 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:05.908645 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:05.908663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.909979 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.909984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:05.911457 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:05.911470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:05.912319 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:05.912327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:05.912585 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:05.913308 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:05.913331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:05.915569 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:05.919766 (+ 4197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:05.919774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:05.919791 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:05.920338 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:05.920793 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:05.920798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:05.920803 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:05.921016 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:05.921603 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:05.921612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:05.922152 (+ 540us) client_negotiation.cc:769] Sending connection context
0927 07:10:05.922491 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 07:10:05.923174 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":266,"threads_started":1}
I20260927 07:10:05.926901 14045 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:05.927259 14045 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:05.928052 14045 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:05.930305 13856 ts_manager.cc:194] Registered new tserver with Master: 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537)
I20260927 07:10:05.931957 13856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:33485
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:10:05.970875 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:06.317922 14051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:06.318300 14051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:06.318475 14051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:06.325989 14051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:06.326179 14051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:06.326373 14051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:10:06.336580 14051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:06.338070 14051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:06.339715 14051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:06.353312 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:06.353258 14057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:06.353286 14059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:06.354899 14051 server_base.cc:1034] running on GCE node
I20260927 07:10:06.355592 14051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:06.356632 14051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:06.357894 14051 hybrid_clock.cc:648] HybridClock initialized: now 1790493006357843 us; error 42 us; skew 500 ppm
I20260927 07:10:06.358253 14051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:06 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:06.361646 14051 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:10:06.363355 14051 webserver.cc:460] Webserver started at http://127.30.16.66:45769/ using document root <none> and password file <none>
I20260927 07:10:06.364095 14051 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:06.364241 14051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:06.364583 14051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:10:06.367882 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "fa9d9110b50449ec9f507a27a581c958"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "acd632b9fc5b46884755ce0fb6dea4f9"
server_key_iv: "4bc1f8c51154bf8e5f712281f7c2b431"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.368767 14051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "fa9d9110b50449ec9f507a27a581c958"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "acd632b9fc5b46884755ce0fb6dea4f9"
server_key_iv: "4bc1f8c51154bf8e5f712281f7c2b431"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.374462 14051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:10:06.378960 14066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.380594 14051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:10:06.380815 14051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "fa9d9110b50449ec9f507a27a581c958"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "acd632b9fc5b46884755ce0fb6dea4f9"
server_key_iv: "4bc1f8c51154bf8e5f712281f7c2b431"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.381045 14051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:06.402055 14051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:06.406777 14051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:06.407097 14051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:06.408134 14051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:06.409490 14051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:06.409591 14051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.409734 14051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:06.409806 14051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.444836 14051 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:39961
I20260927 07:10:06.444883 14178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:39961 every 8 connection(s)
I20260927 07:10:06.446203 14051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:10:06.446496 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14051
I20260927 07:10:06.446749 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:10:06.447371 30785 external_mini_cluster.cc:1428] Setting key 86fc1893d6716ca26d7fe4259cf48ed3
Sep 27 07:10:06 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:06.470055 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:06.449093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43883 (local address 127.30.16.126:43019)
0927 07:10:06.449540 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:06.449545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:06.451043 (+ 1498us) server_negotiation.cc:407] Connection header received
0927 07:10:06.452968 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:06.452978 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:06.453073 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:06.453212 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:06.454868 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.456052 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.458305 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.458825 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.458930 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:06.463705 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:06.463710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:06.463714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:06.463749 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:06.466531 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:06.467589 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:06.467593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:06.467595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:06.467721 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:06.468213 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:06.468216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:06.468218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:06.468484 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:06.468594 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:06.469578 (+ 984us) server_negotiation.cc:299] Negotiation successful
0927 07:10:06.469770 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":243,"threads_started":1}
I20260927 07:10:06.471972 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:06.449952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:43883)
0927 07:10:06.450882 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:06.450930 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:06.452688 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:06.453439 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:06.453448 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:06.454010 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:06.454693 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:06.454713 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.456232 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.456236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:06.458127 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:06.458145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.459073 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.459083 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:06.459440 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:06.460230 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:06.460308 (+ 78us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:06.463428 (+ 3120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:06.466733 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:06.466745 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:06.466767 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:06.467469 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:06.467831 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:06.467839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:06.467846 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:06.468103 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:06.468615 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:06.468627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:06.469147 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 07:10:06.469488 (+ 341us) client_negotiation.cc:240] Negotiation successful
0927 07:10:06.470170 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":373,"threads_started":1}
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
I20260927 07:10:06.473816 14179 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:06.474179 14179 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:06.475124 14179 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:06.476742 13856 ts_manager.cc:194] Registered new tserver with Master: fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
I20260927 07:10:06.477869 13856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:43883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:10:06.507059 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:06.844245 14185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:06.844538 14185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:06.844640 14185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:06.852177 14185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:06.852308 14185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:06.852427 14185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:10:06.862514 14185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:06.863947 14185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:06.865270 14185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:06.879417 14193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:06.879372 14191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:06.879467 14190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:06.884259 14185 server_base.cc:1034] running on GCE node
I20260927 07:10:06.884956 14185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:06.885982 14185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:06.887224 14185 hybrid_clock.cc:648] HybridClock initialized: now 1790493006887153 us; error 59 us; skew 500 ppm
I20260927 07:10:06.887566 14185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:06 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:06.890995 14185 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:10:06.892665 14185 webserver.cc:460] Webserver started at http://127.30.16.67:34355/ using document root <none> and password file <none>
I20260927 07:10:06.893450 14185 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:06.893574 14185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:06.893903 14185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:10:06.897080 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "c00fb34a6d504547b63fa36f866225d7"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "8b790937faabf3bc7e6f82f402eb4d88"
server_key_iv: "98d90f153792c0f9c7c2d67f46cd08cc"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.897879 14185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "c00fb34a6d504547b63fa36f866225d7"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "8b790937faabf3bc7e6f82f402eb4d88"
server_key_iv: "98d90f153792c0f9c7c2d67f46cd08cc"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.903595 14185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:10:06.907284 14200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.908658 14185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:10:06.908825 14185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "c00fb34a6d504547b63fa36f866225d7"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "8b790937faabf3bc7e6f82f402eb4d88"
server_key_iv: "98d90f153792c0f9c7c2d67f46cd08cc"
server_key_version: "encryptionkey@0"
I20260927 07:10:06.909044 14185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:06.924921 14185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:06.929432 14185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:06.929752 14185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:06.930895 14185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:06.932271 14185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:06.932346 14185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.932490 14185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:06.932546 14185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:06.935910 14045 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:06.967247 14185 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:32897
I20260927 07:10:06.967286 14312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:32897 every 8 connection(s)
I20260927 07:10:06.968595 14185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:10:06.971728 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14185
I20260927 07:10:06.971926 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:10:06.972570 30785 external_mini_cluster.cc:1428] Setting key a153231dd081d9965445a8de28c167a2
Sep 27 07:10:06 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493006, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:06.990880 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:06.971312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:35637 (local address 127.30.16.126:43019)
0927 07:10:06.971780 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:06.971785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:06.973370 (+ 1585us) server_negotiation.cc:407] Connection header received
0927 07:10:06.975565 (+ 2195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:06.975570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:06.975679 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:06.975772 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:06.977353 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.978272 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.980001 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.980433 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.980547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:06.984056 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:06.984063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:06.984067 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:06.984105 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:06.987256 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:06.988481 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:06.988491 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:06.988493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:06.988616 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:06.989002 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:06.989005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:06.989007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:06.989301 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:06.989464 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:06.990257 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 07:10:06.990475 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":234,"threads_started":1}
I20260927 07:10:06.991888 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:06.972005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:35637)
0927 07:10:06.972851 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:06.972926 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:06.975231 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:06.976038 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:06.976046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:06.976576 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:06.977164 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:06.977181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.978435 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.978439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:06.979850 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:06.979863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:06.980637 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:06.980642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:06.980846 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:06.981495 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:06.981517 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:06.983828 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:06.987621 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:06.987628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:06.987642 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:06.988343 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:06.988714 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:06.988717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:06.988720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:06.988904 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:06.989430 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:06.989438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:06.989810 (+ 372us) client_negotiation.cc:769] Sending connection context
0927 07:10:06.990141 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 07:10:06.990732 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":311,"threads_started":1}
I20260927 07:10:06.993224 14313 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:06.993637 14313 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:06.994431 14313 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:06.995923 13856 ts_manager.cc:194] Registered new tserver with Master: c00fb34a6d504547b63fa36f866225d7 (127.30.16.67:32897)
I20260927 07:10:06.996742 13856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:35637
I20260927 07:10:07.000881 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:10:07.019341 14315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.005551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34006 (local address 127.30.16.126:43019)
0927 07:10:07.005713 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.005718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.006245 (+ 527us) server_negotiation.cc:407] Connection header received
0927 07:10:07.006402 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.006406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.006485 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.006647 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:07.008253 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.009053 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.010475 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.010967 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.011066 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.013759 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:07.013764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:07.013767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:07.013802 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:07.016697 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.017443 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.017447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.017449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.017570 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.017928 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.017931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.017933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.018271 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:07.018451 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.018908 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.019089 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 07:10:07.023712 13856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:10:07.030673 13856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:10:07.068976 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.043514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:45537 (local address 127.0.0.1:51952)
0927 07:10:07.056033 (+ 12519us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:07.056094 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:07.056395 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:07.059931 (+ 3536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:07.059938 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:07.059979 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:07.060388 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.060403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.062859 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.062865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.065629 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.065649 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.067872 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.067887 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.068112 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.068189 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 07:10:07.068303 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:10:07.068531 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12157,"mutex_wait_us":1,"thread_start_us":11962,"threads_started":1}
I20260927 07:10:07.069253 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.043641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.30.16.67:32897)
0927 07:10:07.055436 (+ 11795us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.055450 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.059364 (+ 3914us) server_negotiation.cc:407] Connection header received
0927 07:10:07.059483 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.059493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.059781 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.059996 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:07.060934 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.062657 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.066605 (+ 3948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.068096 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.068225 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.068466 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.068707 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.068883 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11589,"thread_start_us":11406,"threads_started":1}
I20260927 07:10:07.069276 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.043527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:32897 (local address 127.0.0.1:39368)
0927 07:10:07.059032 (+ 15505us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:07.059055 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:07.059208 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:07.060289 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:07.060294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:07.060319 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:07.060726 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.060743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.062859 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.062865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.066366 (+ 3501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.066380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.068315 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.068321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.068504 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.068537 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 07:10:07.068642 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 07:10:07.068779 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15378,"mutex_wait_us":58,"thread_start_us":22138,"threads_started":1}
I20260927 07:10:07.070573 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.043865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.30.16.65:45537)
0927 07:10:07.058723 (+ 14858us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.058735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.058774 (+ 39us) server_negotiation.cc:407] Connection header received
0927 07:10:07.058900 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.058910 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.059230 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.060670 (+ 1440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:07.060921 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.062639 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.065873 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.067641 (+ 1768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.069674 (+ 2033us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.069868 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.069923 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.070201 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14647,"thread_start_us":14496,"threads_started":1}
I20260927 07:10:07.073446 14248 tablet_service.cc:1467] Processing CreateTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (DEFAULT_TABLE table=test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:07.074193 13980 tablet_service.cc:1467] Processing CreateTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (DEFAULT_TABLE table=test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:07.075012 14248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:07.075695 13980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:07.076972 14327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.056855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39961 (local address 127.0.0.1:46238)
0927 07:10:07.066012 (+ 9157us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:07.066033 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:07.066165 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:07.067189 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:07.067194 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:07.067221 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:07.067672 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.067686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.070125 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.070132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.073953 (+ 3821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.073971 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.076157 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.076163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.076336 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.076371 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:10:07.076473 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 07:10:07.076623 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9008,"thread_start_us":2525,"threads_started":1}
I20260927 07:10:07.077052 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.057099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.30.16.66:39961)
0927 07:10:07.063920 (+ 6821us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.063929 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.066161 (+ 2232us) server_negotiation.cc:407] Connection header received
0927 07:10:07.066321 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.066331 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.066709 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.066923 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:07.067895 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.069973 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.074183 (+ 4210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.075660 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.075810 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.076015 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.076540 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.076724 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6631,"thread_start_us":6289,"threads_started":1}
I20260927 07:10:07.080834 14114 tablet_service.cc:1467] Processing CreateTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (DEFAULT_TABLE table=test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:07.082238 14114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:07.090306 14333 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Bootstrap starting.
I20260927 07:10:07.093585 14332 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Bootstrap starting.
I20260927 07:10:07.095497 14333 tablet_bootstrap.cc:654] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:07.096697 14333 log.cc:824] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:07.096994 14334 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Bootstrap starting.
I20260927 07:10:07.097688 14332 tablet_bootstrap.cc:654] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:07.098999 14332 log.cc:824] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:07.100690 14333 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: No bootstrap required, opened a new log
I20260927 07:10:07.100937 14334 tablet_bootstrap.cc:654] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:07.100979 14333 ts_tablet_manager.cc:1397] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260927 07:10:07.101675 14332 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: No bootstrap required, opened a new log
I20260927 07:10:07.101925 14332 ts_tablet_manager.cc:1397] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 07:10:07.102128 14334 log.cc:824] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:07.109535 14334 tablet_bootstrap.cc:492] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: No bootstrap required, opened a new log
I20260927 07:10:07.109864 14334 ts_tablet_manager.cc:1397] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260927 07:10:07.110044 14332 raft_consensus.cc:348] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.110453 14332 raft_consensus.cc:374] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:07.110584 14332 raft_consensus.cc:729] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00fb34a6d504547b63fa36f866225d7, State: Initialized, Role: FOLLOWER
I20260927 07:10:07.111338 14332 consensus_queue.cc:260] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.112524 14313 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:07.112506 14333 raft_consensus.cc:348] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.112934 14333 raft_consensus.cc:374] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:07.113029 14333 raft_consensus.cc:729] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267dfef0c6224ae2af395d4eda085ddd, State: Initialized, Role: FOLLOWER
I20260927 07:10:07.113245 14332 ts_tablet_manager.cc:1428] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 07:10:07.114032 14333 consensus_queue.cc:260] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.116539 14333 ts_tablet_manager.cc:1428] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 07:10:07.120558 14334 raft_consensus.cc:348] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.120975 14334 raft_consensus.cc:374] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:07.121083 14334 raft_consensus.cc:729] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9d9110b50449ec9f507a27a581c958, State: Initialized, Role: FOLLOWER
I20260927 07:10:07.121656 14334 consensus_queue.cc:260] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.122740 14179 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:07.123272 14334 ts_tablet_manager.cc:1428] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260927 07:10:07.153858 14046 tablet.cc:2369] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:10:07.181401 14338 raft_consensus.cc:482] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:10:07.181706 14338 raft_consensus.cc:504] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.183565 14338 leader_election.cc:288] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537), fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961)
I20260927 07:10:07.190127 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.184004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:45537 (local address 127.30.16.67:39069)
0927 07:10:07.184240 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:07.184265 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:07.184407 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:07.184825 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:07.184830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:07.184860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:07.185129 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.185143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.186349 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.186355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.187946 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.187958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.189355 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.189361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.189570 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.189595 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:10:07.189703 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:10:07.189857 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 07:10:07.190124 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.184146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:39069 (local address 127.30.16.65:45537)
0927 07:10:07.184355 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.184360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.184381 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:10:07.184585 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.184593 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.184682 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.184801 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:07.185309 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.186207 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.188102 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.189195 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.189361 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.189439 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.189730 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.189860 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:10:07.191429 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.184757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:40207 (local address 127.30.16.66:39961)
0927 07:10:07.185028 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.185036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.185534 (+ 498us) server_negotiation.cc:407] Connection header received
0927 07:10:07.185816 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.185820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.185881 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.185981 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:07.186600 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.187407 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.189450 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.190379 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.190534 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.190658 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.191139 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.191245 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 07:10:07.191250 14000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b2c2d2a8a94ad29fb8bf46419ae5be" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267dfef0c6224ae2af395d4eda085ddd" is_pre_election: true
I20260927 07:10:07.191659 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.184514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39961 (local address 127.30.16.67:40207)
0927 07:10:07.185431 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:07.185447 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:07.185592 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:07.185997 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:07.186001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:07.186024 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:07.186389 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.186403 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.187554 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.187560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.189295 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:07.189307 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.190769 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.190778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:07.190949 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.190985 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:10:07.191088 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 07:10:07.191216 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":589,"threads_started":1}
I20260927 07:10:07.191911 14000 raft_consensus.cc:2454] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 0.
I20260927 07:10:07.192641 14202 leader_election.cc:302] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267dfef0c6224ae2af395d4eda085ddd, c00fb34a6d504547b63fa36f866225d7; no voters:
I20260927 07:10:07.192689 14134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b2c2d2a8a94ad29fb8bf46419ae5be" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9d9110b50449ec9f507a27a581c958" is_pre_election: true
I20260927 07:10:07.193080 14338 raft_consensus.cc:2781] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:10:07.193244 14338 raft_consensus.cc:482] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:10:07.193207 14134 raft_consensus.cc:2454] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 0.
I20260927 07:10:07.193354 14338 raft_consensus.cc:3037] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:07.195765 14338 raft_consensus.cc:504] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.196316 14338 leader_election.cc:288] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 election: Requested vote from peers 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537), fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961)
I20260927 07:10:07.196851 14000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b2c2d2a8a94ad29fb8bf46419ae5be" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267dfef0c6224ae2af395d4eda085ddd"
I20260927 07:10:07.196897 14134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b2c2d2a8a94ad29fb8bf46419ae5be" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9d9110b50449ec9f507a27a581c958"
I20260927 07:10:07.197090 14000 raft_consensus.cc:3037] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:07.197090 14134 raft_consensus.cc:3037] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Advancing to term 1
W20260927 07:10:07.198763 14180 tablet.cc:2369] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:10:07.200423 14000 raft_consensus.cc:2454] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 1.
I20260927 07:10:07.200423 14134 raft_consensus.cc:2454] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 1.
I20260927 07:10:07.200834 14202 leader_election.cc:302] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267dfef0c6224ae2af395d4eda085ddd, c00fb34a6d504547b63fa36f866225d7; no voters:
I20260927 07:10:07.201102 14338 raft_consensus.cc:2781] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:10:07.201658 14338 raft_consensus.cc:686] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 1 LEADER]: Becoming Leader. State: Replica: c00fb34a6d504547b63fa36f866225d7, State: Running, Role: LEADER
I20260927 07:10:07.202185 14338 consensus_queue.cc:237] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } }
I20260927 07:10:07.206955 13855 catalog_manager.cc:5495] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c00fb34a6d504547b63fa36f866225d7 (127.30.16.67). New cstate: current_term: 1 leader_uuid: "c00fb34a6d504547b63fa36f866225d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } health_report { overall_health: UNKNOWN } } }
W20260927 07:10:07.220263 14314 tablet.cc:2369] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:10:07.232236 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.225911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.30.16.67:32897)
0927 07:10:07.226093 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.226097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.226129 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:10:07.226357 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.226363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.226461 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.226651 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:10:07.227076 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.228163 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.229670 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.230302 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.230421 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.230982 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:10:07.231679 (+ 697us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:10:07.231821 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.231874 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.231956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 07:10:07.242024 14134 raft_consensus.cc:1261] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Refusing update from remote peer c00fb34a6d504547b63fa36f866225d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:10:07.242063 14000 raft_consensus.cc:1261] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Refusing update from remote peer c00fb34a6d504547b63fa36f866225d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:10:07.243075 14342 consensus_queue.cc:1035] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:10:07.243460 14338 consensus_queue.cc:1035] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:10:07.255679 14345 mvcc.cc:205] Tried to move back new op lower bound from 7333859357652557824 to 7333859357504667648. Current Snapshot: MvccSnapshot[applied={T|T < 7333859357652557824}]
I20260927 07:10:07.269351 13856 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:10:07.269629 13856 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:10:07.273414 13856 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Sending DeleteTablet for 3 replicas of tablet 39b2c2d2a8a94ad29fb8bf46419ae5be
I20260927 07:10:07.274749 14248 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:07 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:39368
I20260927 07:10:07.274791 13980 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:07 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:51952
I20260927 07:10:07.275197 14114 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b2c2d2a8a94ad29fb8bf46419ae5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:07 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:46238
I20260927 07:10:07.276503 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13917
I20260927 07:10:07.304483 14356 tablet_replica.cc:330] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: stopping tablet replica
I20260927 07:10:07.304807 14357 tablet_replica.cc:330] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: stopping tablet replica
I20260927 07:10:07.305054 14356 raft_consensus.cc:2229] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:10:07.305454 14357 raft_consensus.cc:2229] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:10:07.305673 14356 raft_consensus.cc:2258] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:07.306123 14357 raft_consensus.cc:2258] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:07.307112 14355 tablet_replica.cc:330] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: stopping tablet replica
I20260927 07:10:07.307559 14355 raft_consensus.cc:2229] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:10:07.308257 14355 raft_consensus.cc:2258] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:07.309734 14356 ts_tablet_manager.cc:1905] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:07.310424 14355 ts_tablet_manager.cc:1905] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:07.314410 14357 ts_tablet_manager.cc:1905] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:07.317543 14356 ts_tablet_manager.cc:1918] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:07.317762 14356 log.cc:1196] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/39b2c2d2a8a94ad29fb8bf46419ae5be
I20260927 07:10:07.318300 14356 ts_tablet_manager.cc:1939] T 39b2c2d2a8a94ad29fb8bf46419ae5be P 267dfef0c6224ae2af395d4eda085ddd: Deleting consensus metadata
I20260927 07:10:07.318760 14355 ts_tablet_manager.cc:1918] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:07.318957 14355 log.cc:1196] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/39b2c2d2a8a94ad29fb8bf46419ae5be
I20260927 07:10:07.319553 14355 ts_tablet_manager.cc:1939] T 39b2c2d2a8a94ad29fb8bf46419ae5be P c00fb34a6d504547b63fa36f866225d7: Deleting consensus metadata
I20260927 07:10:07.321630 13842 catalog_manager.cc:4841] TS c00fb34a6d504547b63fa36f866225d7 (127.30.16.67:32897): tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (table test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]) successfully deleted
I20260927 07:10:07.321691 14357 ts_tablet_manager.cc:1918] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:07.321857 14357 log.cc:1196] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/39b2c2d2a8a94ad29fb8bf46419ae5be
I20260927 07:10:07.322388 14357 ts_tablet_manager.cc:1939] T 39b2c2d2a8a94ad29fb8bf46419ae5be P fa9d9110b50449ec9f507a27a581c958: Deleting consensus metadata
I20260927 07:10:07.324081 13841 catalog_manager.cc:4841] TS 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537): tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (table test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]) successfully deleted
I20260927 07:10:07.324586 13841 catalog_manager.cc:4841] TS fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961): tablet 39b2c2d2a8a94ad29fb8bf46419ae5be (table test-table [id=6a3670dbbc3b43f0aa2538abef4f4fef]) successfully deleted
Sep 27 07:10:07 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:10:07.333709 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.278070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.30.16.65:45537)
0927 07:10:07.278320 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.278330 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.301483 (+ 23153us) server_negotiation.cc:407] Connection header received
0927 07:10:07.302231 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.302242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.302388 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.302491 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:07.307689 (+ 5198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.308850 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.320823 (+ 11973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.321480 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.321607 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.327577 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:07.327590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:07.327608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:07.327647 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:07.330679 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.331447 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.331455 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.331458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.331599 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.332015 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.332018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.332020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.332423 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:07.332757 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.333089 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.333326 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 07:10:07.334436 14040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:07.428841 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13917
I20260927 07:10:07.451038 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14051
Sep 27 07:10:07 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:10:07.472002 14331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.451892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.30.16.66:39961)
0927 07:10:07.452123 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.452130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.452151 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:10:07.452325 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.452331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.452431 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.452542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:07.454264 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.455554 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.457548 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.458234 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.458359 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.464453 (+ 6094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:07.464466 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:07.464486 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:07.464530 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:07.468611 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.469543 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.469553 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.469557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.469714 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.470161 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.470165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.470168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.470726 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:07.471024 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.471380 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.471610 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 07:10:07.472759 14174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:07.575673 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14051
I20260927 07:10:07.594202 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14185
Sep 27 07:10:07 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:10:07.612409 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:07.595341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.30.16.67:32897)
0927 07:10:07.595558 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:07.595565 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:07.595585 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:10:07.595660 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:07.595665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:07.595761 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:07.595899 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:07.597202 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.598422 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.599914 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:07.600623 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:07.600796 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:07.605604 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:07.605618 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:07.605638 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:07.605686 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:07.608994 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.609919 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.609926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.609929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.610089 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:07.610743 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:07.610746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:07.610748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:07.611188 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:07.611416 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:07.611790 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 07:10:07.612000 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 07:10:07.613245 14308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:07.740388 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14185
I20260927 07:10:07.757824 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 13824
I20260927 07:10:07.759136 13886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:07.872535 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 13824
I20260927 07:10:27.894793 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--webserver_interface=127.30.16.126
--webserver_port=38979
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:28.241405 14363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:28.241710 14363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:28.241880 14363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:28.249421 14363 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:10:28.249519 14363 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:10:28.249586 14363 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:10:28.249673 14363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:10:28.249794 14363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:28.249917 14363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:28.249995 14363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:10:28.250105 14363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:10:28.260536 14363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:28.261996 14363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:28.263968 14363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:28.287788 14369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:28.287725 14372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:28.287786 14370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:28.288708 14363 server_base.cc:1034] running on GCE node
I20260927 07:10:28.289611 14363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:28.291359 14363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:28.292586 14363 hybrid_clock.cc:648] HybridClock initialized: now 1790493028292508 us; error 50 us; skew 500 ppm
I20260927 07:10:28.293009 14363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:28 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493028, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:28.297057 14363 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:10:28.298995 14363 webserver.cc:460] Webserver started at http://127.30.16.126:38979/ using document root <none> and password file <none>
I20260927 07:10:28.299844 14363 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:28.300002 14363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:28.306943 14363 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:10:28.310410 14379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:28.312043 14363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:28.312260 14363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5be94b8ece764074bd4abe5e970e814d"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "3fd7556322451c661933538e4207a431"
server_key_iv: "9580087c1e871de9cfe3356c818ff6ac"
server_key_version: "encryptionkey@0"
I20260927 07:10:28.313400 14363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:28.328429 14363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:28.333123 14363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:28.333447 14363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:28.354064 14363 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43019
I20260927 07:10:28.354156 14430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43019 every 8 connection(s)
I20260927 07:10:28.355831 14363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:10:28.360278 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14363
I20260927 07:10:28.361156 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:45537
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=45339
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:10:28.364073 14431 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:10:28.367830 14431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap starting.
I20260927 07:10:28.395401 14431 log.cc:824] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:28.407907 14431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:10:28.408493 14431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap complete.
I20260927 07:10:28.418345 14431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:28.419296 14431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Initialized, Role: FOLLOWER
I20260927 07:10:28.420061 14431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:28.420342 14431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:10:28.420469 14431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:10:28.420665 14431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 1 FOLLOWER]: Advancing to term 2
I20260927 07:10:28.424790 14431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:28.425375 14431 leader_election.cc:302] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5be94b8ece764074bd4abe5e970e814d; no voters:
I20260927 07:10:28.426048 14431 leader_election.cc:288] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 07:10:28.426277 14435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 07:10:28.427472 14435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 LEADER]: Becoming Leader. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Running, Role: LEADER
I20260927 07:10:28.428292 14431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:10:28.428192 14435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:28.430963 14437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be94b8ece764074bd4abe5e970e814d. Latest consensus state: current_term: 2 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:28.431432 14437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:28.432039 14436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:28.432359 14436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:28.440011 14442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:10:28.448992 14442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:10:28.450587 14442 catalog_manager.cc:1259] Loaded cluster ID: abf825cee14240ea8b4208f34af1ee70
I20260927 07:10:28.450753 14442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:10:28.455093 14442 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:10:28.456717 14442 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Loaded TSK: 0
I20260927 07:10:28.491204 14442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Generated new TSK 1
I20260927 07:10:28.491639 14442 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 07:10:28.737695 14433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:28.738036 14433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:28.738224 14433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:28.745841 14433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:28.746027 14433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:28.746238 14433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:10:28.756844 14433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:45537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=45339
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:28.758272 14433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:28.759784 14433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:28.774803 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:28.774811 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:28.775264 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:28.775985 14433 server_base.cc:1034] running on GCE node
I20260927 07:10:28.776676 14433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:28.777760 14433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:28.778975 14433 hybrid_clock.cc:648] HybridClock initialized: now 1790493028778907 us; error 58 us; skew 500 ppm
I20260927 07:10:28.779315 14433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:28 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493028, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:28.782719 14433 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:10:28.784317 14433 webserver.cc:460] Webserver started at http://127.30.16.65:45339/ using document root <none> and password file <none>
I20260927 07:10:28.785084 14433 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:28.785255 14433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:28.791201 14433 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:10:28.794099 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:28.795745 14433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:28.795928 14433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "267dfef0c6224ae2af395d4eda085ddd"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "0590c87ef2eb0ef1defad69f0205fd12"
server_key_iv: "8cceb54977e74f45f95c966fcad544bc"
server_key_version: "encryptionkey@0"
I20260927 07:10:28.796907 14433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:28.830613 14433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:28.835405 14433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:28.835703 14433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:28.836725 14433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:28.838112 14433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:28.838209 14433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:28.838357 14433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:28.838457 14433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:28.871515 14433 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45537
I20260927 07:10:28.871642 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45537 every 8 connection(s)
I20260927 07:10:28.873041 14433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:10:28.882244 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14433
I20260927 07:10:28.883098 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:39961
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=45769
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:10:28 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493028, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:28.899664 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:28.876123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:54625 (local address 127.30.16.126:43019)
0927 07:10:28.876729 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:28.876743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:28.877368 (+ 625us) server_negotiation.cc:407] Connection header received
0927 07:10:28.879594 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:28.879607 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:28.879947 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:28.880261 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:28.881677 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:28.883896 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:28.885518 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:28.886679 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:28.886883 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:28.890685 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:28.890698 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:28.890724 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:28.890786 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:28.894123 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:28.895536 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:28.895544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:28.895550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:28.895739 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:28.896354 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:28.896357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:28.896359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:28.896881 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:28.897090 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:28.898197 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0927 07:10:28.898527 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260927 07:10:28.900398 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:28.876375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:54625)
0927 07:10:28.877202 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:28.877248 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:28.879059 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:28.880395 (+ 1336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:28.880402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:28.880940 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:28.881475 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:28.881491 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:28.884095 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:28.884099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:28.885364 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:28.885376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:28.886894 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:28.886899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:28.887106 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:28.887829 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:28.887852 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:28.890316 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:28.894309 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:28.894324 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:28.894347 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:28.895364 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:28.895890 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:28.895900 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:28.895907 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:28.896234 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:28.897082 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:28.897096 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:28.897636 (+ 540us) client_negotiation.cc:769] Sending connection context
0927 07:10:28.898022 (+ 386us) client_negotiation.cc:240] Negotiation successful
0927 07:10:28.898761 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":358,"threads_started":1}
I20260927 07:10:28.904523 14582 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:28.904899 14582 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:28.905707 14582 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:28.908726 14396 ts_manager.cc:194] Registered new tserver with Master: 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537)
I20260927 07:10:28.911516 14396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:54625
W20260927 07:10:29.239639 14586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:29.239945 14586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:29.240042 14586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:29.247395 14586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:29.247516 14586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:29.247630 14586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:10:29.257715 14586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:39961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=45769
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:29.259019 14586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:29.260306 14586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:29.273919 14591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:29.274052 14594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:29.273947 14592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:29.274860 14586 server_base.cc:1034] running on GCE node
I20260927 07:10:29.275493 14586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:29.276515 14586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:29.277751 14586 hybrid_clock.cc:648] HybridClock initialized: now 1790493029277684 us; error 62 us; skew 500 ppm
I20260927 07:10:29.278082 14586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:29 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493029, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:29.281303 14586 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:10:29.282897 14586 webserver.cc:460] Webserver started at http://127.30.16.66:45769/ using document root <none> and password file <none>
I20260927 07:10:29.283708 14586 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:29.283864 14586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:29.289417 14586 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 07:10:29.292177 14601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.293598 14586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:10:29.293735 14586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "fa9d9110b50449ec9f507a27a581c958"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "acd632b9fc5b46884755ce0fb6dea4f9"
server_key_iv: "4bc1f8c51154bf8e5f712281f7c2b431"
server_key_version: "encryptionkey@0"
I20260927 07:10:29.294732 14586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:29.319945 14586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:29.324505 14586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:29.324739 14586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:29.325690 14586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:29.327142 14586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:29.327229 14586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.327306 14586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:29.327360 14586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.362295 14586 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:39961
I20260927 07:10:29.362347 14713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:39961 every 8 connection(s)
I20260927 07:10:29.363862 14586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:10:29.373029 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14586
I20260927 07:10:29.373840 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:32897
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=34355
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 07:10:29 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493029, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:29.388463 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.366780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:50257 (local address 127.30.16.126:43019)
0927 07:10:29.366991 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.366995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.368394 (+ 1399us) server_negotiation.cc:407] Connection header received
0927 07:10:29.370298 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.370305 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.370407 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.370515 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:29.372070 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.372836 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.375398 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.375871 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.375975 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.381010 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:29.381016 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:29.381019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:29.381055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:29.384531 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.385447 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.385456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.385463 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.385630 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.386226 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.386231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.386236 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.386698 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:29.386876 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.387706 (+ 830us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.388007 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:10:29.389266 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.367365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:50257)
0927 07:10:29.368218 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:29.368274 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:29.370019 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:29.370785 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:29.370793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:29.371317 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:29.371879 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.371895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.373013 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.373019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.375225 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.375244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.376116 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.376125 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.376482 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.377306 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:29.377451 (+ 145us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:29.380730 (+ 3279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:29.384714 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:29.384722 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:29.384737 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:29.385302 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:29.385780 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:29.385785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:29.385787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:29.386028 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:29.386873 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:29.386889 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:29.387291 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 07:10:29.387552 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 07:10:29.388008 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":308,"threads_started":1}
I20260927 07:10:29.390955 14714 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:29.391317 14714 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:29.392199 14714 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:29.393967 14396 ts_manager.cc:194] Registered new tserver with Master: fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961)
I20260927 07:10:29.394946 14396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:50257
W20260927 07:10:29.723984 14717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:29.724287 14717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:29.724385 14717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:29.731976 14717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:29.732095 14717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:29.732285 14717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:10:29.742358 14717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:32897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=34355
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:29.743752 14717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:29.745072 14717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:29.758952 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:29.758988 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:29.759233 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:29.759745 14717 server_base.cc:1034] running on GCE node
I20260927 07:10:29.760370 14717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:29.761420 14717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:29.762686 14717 hybrid_clock.cc:648] HybridClock initialized: now 1790493029762600 us; error 73 us; skew 500 ppm
I20260927 07:10:29.763051 14717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:29 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493029, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:29.766639 14717 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:10:29.768181 14717 webserver.cc:460] Webserver started at http://127.30.16.67:34355/ using document root <none> and password file <none>
I20260927 07:10:29.768927 14717 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:29.769114 14717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:29.775065 14717 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 07:10:29.778079 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.779654 14717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:29.779870 14717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "c00fb34a6d504547b63fa36f866225d7"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "8b790937faabf3bc7e6f82f402eb4d88"
server_key_iv: "98d90f153792c0f9c7c2d67f46cd08cc"
server_key_version: "encryptionkey@0"
I20260927 07:10:29.781036 14717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:29.814356 14717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:29.819413 14717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:29.819712 14717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:29.820752 14717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:29.822134 14717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:29.822288 14717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.822434 14717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:29.822527 14717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:29.857431 14717 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:32897
I20260927 07:10:29.857486 14844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:32897 every 8 connection(s)
I20260927 07:10:29.858966 14717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:10:29.864398 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14717
Sep 27 07:10:29 dist-test-slave-8vqf krb5kdc[13811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493029, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:10:29.884040 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.865454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50756 (local address 127.30.16.126:43019)
0927 07:10:29.866056 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.866062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.867177 (+ 1115us) server_negotiation.cc:407] Connection header received
0927 07:10:29.867316 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.867321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.867404 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.867518 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:29.869816 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.871020 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.872672 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.873294 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.873416 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.876956 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:29.876964 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:29.876969 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:29.877015 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:29.880631 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.881479 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.881488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.881495 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.881685 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.882105 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.882110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.882113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.882535 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:29.882835 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.883279 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.883540 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":321,"threads_started":1}
I20260927 07:10:29.884879 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.861676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:34517 (local address 127.30.16.126:43019)
0927 07:10:29.861876 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.861881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.863131 (+ 1250us) server_negotiation.cc:407] Connection header received
0927 07:10:29.865162 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.865168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.865303 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.865437 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:29.867634 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.869115 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.871562 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.872350 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.872508 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.877459 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:29.877464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:29.877467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:29.877510 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:29.880778 (+ 3268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.881767 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.881773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.881776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.881923 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.882517 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.882520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.882522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.882955 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:29.883106 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.884230 (+ 1124us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.884464 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:10:29.887128 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.862082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:34517)
0927 07:10:29.862894 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:29.862934 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:29.864794 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:29.865771 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:29.865787 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:29.866495 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:10:29.867295 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.867323 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.869314 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.869318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.871381 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.871413 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.872631 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.872641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.872972 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.873886 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:10:29.873924 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:10:29.877201 (+ 3277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:10:29.880933 (+ 3732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:29.880942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:29.880957 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:29.881587 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:29.882120 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:10:29.882128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:10:29.882131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:10:29.882385 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:10:29.883215 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:10:29.883225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:10:29.883733 (+ 508us) client_negotiation.cc:769] Sending connection context
0927 07:10:29.884091 (+ 358us) client_negotiation.cc:240] Negotiation successful
0927 07:10:29.885527 (+ 1436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":288,"threads_started":1}
I20260927 07:10:29.888590 14845 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43019
I20260927 07:10:29.888939 14845 heartbeater.cc:464] Registering TS with master...
I20260927 07:10:29.889698 14845 heartbeater.cc:511] Master 127.30.16.126:43019 requested a full tablet report, sending...
I20260927 07:10:29.891244 14396 ts_manager.cc:194] Registered new tserver with Master: c00fb34a6d504547b63fa36f866225d7 (127.30.16.67:32897)
I20260927 07:10:29.892051 14396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:34517
I20260927 07:10:29.900039 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:10:29.906932 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.900946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.30.16.126:43019)
0927 07:10:29.901216 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.901221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.901673 (+ 452us) server_negotiation.cc:407] Connection header received
0927 07:10:29.901861 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.901865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.902033 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.902168 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:10:29.902669 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.903478 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.905007 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.905419 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.905522 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.906057 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:10:29.906288 (+ 231us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 07:10:29.906674 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50758: token expired
Metrics: {"server-negotiator.queue_time_us":142}
W20260927 07:10:29.907091 14584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50758: token expired
W20260927 07:10:29.907605 14319 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.30.16.126:43019)
I20260927 07:10:29.915616 14582 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:29.921921 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.908093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50764 (local address 127.30.16.126:43019)
0927 07:10:29.908263 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.908268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.908386 (+ 118us) server_negotiation.cc:407] Connection header received
0927 07:10:29.908567 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.908572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.908649 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.908759 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:29.910227 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.911081 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.912578 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.912991 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.913125 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.916209 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:29.916215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:29.916222 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:29.916264 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:29.919351 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.920084 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.920088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.920090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.920206 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:29.920508 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:29.920511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:29.920513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:29.920834 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:29.921016 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.921379 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.921548 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:10:29.925542 14396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:10:29.933169 14396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:10:29.957106 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.946710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39150 (local address 127.30.16.67:32897)
0927 07:10:29.948324 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.948333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.948364 (+ 31us) server_negotiation.cc:407] Connection header received
0927 07:10:29.948457 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.948464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.948718 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.949573 (+ 855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:29.949832 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.951449 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.954636 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.955871 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956022 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.956274 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.956551 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.956723 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1421,"thread_start_us":439,"threads_started":1}
I20260927 07:10:29.957151 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.946191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:32897 (local address 127.0.0.1:39150)
0927 07:10:29.947679 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:29.947700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:29.947832 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:29.948934 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:29.948941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:29.948967 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:29.949377 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.949389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.951612 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.951617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.954431 (+ 2814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.954446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.956092 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.956295 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.956350 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:10:29.956507 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 07:10:29.956737 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1348,"mutex_wait_us":127,"thread_start_us":1793,"threads_started":1}
I20260927 07:10:29.957710 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.947807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.30.16.65:45537)
0927 07:10:29.949066 (+ 1259us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.949074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.949106 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:10:29.949176 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.949181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.949335 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.949524 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:29.950117 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.951789 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.954864 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.956426 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956888 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.957121 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.957189 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.957376 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":927,"threads_started":1}
I20260927 07:10:29.957906 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.945651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52140 (local address 127.30.16.66:39961)
0927 07:10:29.946305 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:29.946315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:29.946846 (+ 531us) server_negotiation.cc:407] Connection header received
0927 07:10:29.947352 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:29.947361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:29.947637 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:29.947829 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:29.949065 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.950956 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.954628 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.956087 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.956431 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:29.957404 (+ 973us) server_negotiation.cc:299] Negotiation successful
0927 07:10:29.957581 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
I20260927 07:10:29.957965 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.947498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:45537 (local address 127.0.0.1:45446)
0927 07:10:29.948373 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:29.948388 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:29.948494 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:29.949506 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:29.949510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:29.949534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:29.949862 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.949914 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.951916 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.951924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.954177 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.954234 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.956671 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.956868 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.956905 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:10:29.956984 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 07:10:29.957120 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":112,"thread_start_us":595,"threads_started":1}
I20260927 07:10:29.958613 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:29.945544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39961 (local address 127.0.0.1:52140)
0927 07:10:29.946631 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:29.946689 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:29.947096 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:29.948334 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:29.948344 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:29.948381 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:29.948823 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.948838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.951115 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.951121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.954439 (+ 3318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:29.954447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:29.956615 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:29.956622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:29.957171 (+ 549us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:29.957206 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 07:10:29.958095 (+ 889us) client_negotiation.cc:240] Negotiation successful
0927 07:10:29.958266 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":186,"thread_start_us":701,"threads_started":1}
I20260927 07:10:29.961582 14780 tablet_service.cc:1467] Processing CreateTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 (DEFAULT_TABLE table=test-table [id=94938fccc635407fb0a6a9ce903c4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:29.961782 14517 tablet_service.cc:1467] Processing CreateTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 (DEFAULT_TABLE table=test-table [id=94938fccc635407fb0a6a9ce903c4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:29.962061 14649 tablet_service.cc:1467] Processing CreateTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 (DEFAULT_TABLE table=test-table [id=94938fccc635407fb0a6a9ce903c4c34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:10:29.962984 14780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d171635f93b403d929b3f18ca0d34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:29.963285 14649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d171635f93b403d929b3f18ca0d34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:29.963286 14517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d171635f93b403d929b3f18ca0d34f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:10:29.975157 14861 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Bootstrap starting.
I20260927 07:10:29.979210 14861 tablet_bootstrap.cc:654] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:29.979676 14863 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Bootstrap starting.
I20260927 07:10:29.980721 14861 log.cc:824] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:29.982167 14862 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Bootstrap starting.
I20260927 07:10:29.983650 14863 tablet_bootstrap.cc:654] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:29.984051 14861 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: No bootstrap required, opened a new log
I20260927 07:10:29.984332 14861 ts_tablet_manager.cc:1397] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 07:10:29.984962 14863 log.cc:824] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:29.985914 14862 tablet_bootstrap.cc:654] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Neither blocks nor log segments found. Creating new log.
I20260927 07:10:29.987239 14862 log.cc:824] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:29.988987 14863 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: No bootstrap required, opened a new log
I20260927 07:10:29.989336 14863 ts_tablet_manager.cc:1397] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 07:10:29.992702 14861 raft_consensus.cc:348] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:29.993180 14861 raft_consensus.cc:374] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:29.993307 14861 raft_consensus.cc:729] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c00fb34a6d504547b63fa36f866225d7, State: Initialized, Role: FOLLOWER
I20260927 07:10:29.993966 14861 consensus_queue.cc:260] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:29.995087 14845 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:29.995175 14862 tablet_bootstrap.cc:492] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: No bootstrap required, opened a new log
I20260927 07:10:29.995453 14862 ts_tablet_manager.cc:1397] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260927 07:10:29.995765 14861 ts_tablet_manager.cc:1428] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Time spent starting tablet: real 0.011s user 0.004s sys 0.008s
I20260927 07:10:30.000666 14863 raft_consensus.cc:348] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.001152 14863 raft_consensus.cc:374] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:30.001256 14863 raft_consensus.cc:729] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 267dfef0c6224ae2af395d4eda085ddd, State: Initialized, Role: FOLLOWER
I20260927 07:10:30.002138 14863 consensus_queue.cc:260] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.004379 14863 ts_tablet_manager.cc:1428] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260927 07:10:30.006958 14862 raft_consensus.cc:348] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.007413 14862 raft_consensus.cc:374] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:10:30.007546 14862 raft_consensus.cc:729] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa9d9110b50449ec9f507a27a581c958, State: Initialized, Role: FOLLOWER
I20260927 07:10:30.008152 14862 consensus_queue.cc:260] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.009310 14714 heartbeater.cc:503] Master 127.30.16.126:43019 was elected leader, sending a full tablet report...
I20260927 07:10:30.009929 14862 ts_tablet_manager.cc:1428] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260927 07:10:30.110976 14846 tablet.cc:2369] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:10:30.116080 14715 tablet.cc:2369] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:10:30.125622 14583 tablet.cc:2369] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:10:30.333038 14867 raft_consensus.cc:482] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:10:30.333380 14867 raft_consensus.cc:504] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.335202 14867 leader_election.cc:288] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961), 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537)
I20260927 07:10:30.341565 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.335629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:53109 (local address 127.30.16.66:39961)
0927 07:10:30.335855 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.335862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.335882 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:10:30.336424 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.336430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.336520 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.336632 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:30.337060 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.337777 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.339478 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.340599 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.340757 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.340833 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.341145 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.341289 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:10:30.341605 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.335503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39961 (local address 127.30.16.67:53109)
0927 07:10:30.335709 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:30.335741 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:30.335886 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:30.336695 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:30.336700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:30.336727 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:30.336918 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:30.336927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.337925 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.337931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:30.339300 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:30.339312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.340756 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.340761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:30.340943 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.340994 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 07:10:30.341088 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 07:10:30.341251 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":1}
I20260927 07:10:30.342317 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.336182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:33619 (local address 127.30.16.65:45537)
0927 07:10:30.336453 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.336459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.336611 (+ 152us) server_negotiation.cc:407] Connection header received
0927 07:10:30.336790 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.336795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.336864 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.336960 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:10:30.337519 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.338235 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.340098 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.341256 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.341411 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.341481 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.341943 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.342071 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 07:10:30.342358 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.335959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:45537 (local address 127.30.16.67:33619)
0927 07:10:30.336508 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:30.336522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:10:30.336640 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:10:30.336972 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:10:30.336976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:10:30.336999 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:10:30.337346 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:30.337354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.338373 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.338380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:30.339940 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:10:30.339952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.341486 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.341492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:10:30.341701 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.341733 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:10:30.341868 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 07:10:30.341993 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":18,"thread_start_us":295,"threads_started":1}
I20260927 07:10:30.342672 14669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d171635f93b403d929b3f18ca0d34f1" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9d9110b50449ec9f507a27a581c958" is_pre_election: true
I20260927 07:10:30.343264 14669 raft_consensus.cc:2454] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 0.
I20260927 07:10:30.343392 14537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d171635f93b403d929b3f18ca0d34f1" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267dfef0c6224ae2af395d4eda085ddd" is_pre_election: true
I20260927 07:10:30.343977 14537 raft_consensus.cc:2454] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 0.
I20260927 07:10:30.344010 14734 leader_election.cc:302] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c00fb34a6d504547b63fa36f866225d7, fa9d9110b50449ec9f507a27a581c958; no voters:
I20260927 07:10:30.344466 14867 raft_consensus.cc:2781] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:10:30.344659 14867 raft_consensus.cc:482] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:10:30.344812 14867 raft_consensus.cc:3037] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:30.347147 14867 raft_consensus.cc:504] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.347779 14867 leader_election.cc:288] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 election: Requested vote from peers fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961), 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537)
I20260927 07:10:30.348300 14537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d171635f93b403d929b3f18ca0d34f1" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "267dfef0c6224ae2af395d4eda085ddd"
I20260927 07:10:30.348300 14669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d171635f93b403d929b3f18ca0d34f1" candidate_uuid: "c00fb34a6d504547b63fa36f866225d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa9d9110b50449ec9f507a27a581c958"
I20260927 07:10:30.348519 14669 raft_consensus.cc:3037] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:30.348521 14537 raft_consensus.cc:3037] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:10:30.351805 14669 raft_consensus.cc:2454] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 1.
I20260927 07:10:30.351805 14537 raft_consensus.cc:2454] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c00fb34a6d504547b63fa36f866225d7 in term 1.
I20260927 07:10:30.352236 14734 leader_election.cc:302] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 267dfef0c6224ae2af395d4eda085ddd, c00fb34a6d504547b63fa36f866225d7; no voters:
I20260927 07:10:30.352548 14867 raft_consensus.cc:2781] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:10:30.353044 14867 raft_consensus.cc:686] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 1 LEADER]: Becoming Leader. State: Replica: c00fb34a6d504547b63fa36f866225d7, State: Running, Role: LEADER
I20260927 07:10:30.353632 14867 consensus_queue.cc:237] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } }
I20260927 07:10:30.359144 14395 catalog_manager.cc:5495] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c00fb34a6d504547b63fa36f866225d7 (127.30.16.67). New cstate: current_term: 1 leader_uuid: "c00fb34a6d504547b63fa36f866225d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c00fb34a6d504547b63fa36f866225d7" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 32897 } health_report { overall_health: HEALTHY } } }
I20260927 07:10:30.416113 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.410870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.30.16.67:32897)
0927 07:10:30.411067 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.411071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.411091 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:10:30.411300 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.411304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.411382 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.411514 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:10:30.411869 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.412596 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.413926 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.414339 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.414430 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.415038 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:10:30.415552 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:10:30.415666 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.415843 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.415910 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:10:30.424751 14537 raft_consensus.cc:1261] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Refusing update from remote peer c00fb34a6d504547b63fa36f866225d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:10:30.424751 14669 raft_consensus.cc:1261] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Refusing update from remote peer c00fb34a6d504547b63fa36f866225d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:10:30.425817 14871 consensus_queue.cc:1035] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "267dfef0c6224ae2af395d4eda085ddd" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 45537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:10:30.426258 14867 consensus_queue.cc:1035] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa9d9110b50449ec9f507a27a581c958" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:10:30.439920 14874 mvcc.cc:205] Tried to move back new op lower bound from 7333859452609007616 to 7333859452333412352. Current Snapshot: MvccSnapshot[applied={T|T < 7333859452609007616}]
I20260927 07:10:30.441290 14875 mvcc.cc:205] Tried to move back new op lower bound from 7333859452609007616 to 7333859452333412352. Current Snapshot: MvccSnapshot[applied={T|T < 7333859452609007616}]
I20260927 07:10:30.453840 14395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:10:30.454118 14395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:10:30.457978 14395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Sending DeleteTablet for 3 replicas of tablet 0d171635f93b403d929b3f18ca0d34f1
I20260927 07:10:30.459555 14517 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:45446
I20260927 07:10:30.459555 14780 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:39150
I20260927 07:10:30.459546 14649 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d171635f93b403d929b3f18ca0d34f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:10:30 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:52140
I20260927 07:10:30.459995 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14433
I20260927 07:10:30.471613 14883 tablet_replica.cc:330] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: stopping tablet replica
I20260927 07:10:30.472266 14883 raft_consensus.cc:2229] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:10:30.472909 14883 raft_consensus.cc:2258] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:30.476001 14883 ts_tablet_manager.cc:1905] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:30.477399 14884 tablet_replica.cc:330] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: stopping tablet replica
I20260927 07:10:30.477887 14884 raft_consensus.cc:2229] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:10:30.478439 14884 raft_consensus.cc:2258] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:30.479136 14882 tablet_replica.cc:330] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: stopping tablet replica
I20260927 07:10:30.479620 14882 raft_consensus.cc:2229] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:10:30.480414 14882 raft_consensus.cc:2258] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:10:30.482312 14884 ts_tablet_manager.cc:1905] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:30.482784 14882 ts_tablet_manager.cc:1905] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:10:30.488529 14883 ts_tablet_manager.cc:1918] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:30.488845 14883 log.cc:1196] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/0d171635f93b403d929b3f18ca0d34f1
I20260927 07:10:30.489696 14883 ts_tablet_manager.cc:1939] T 0d171635f93b403d929b3f18ca0d34f1 P 267dfef0c6224ae2af395d4eda085ddd: Deleting consensus metadata
I20260927 07:10:30.491009 14882 ts_tablet_manager.cc:1918] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:30.491322 14882 log.cc:1196] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/0d171635f93b403d929b3f18ca0d34f1
I20260927 07:10:30.492049 14882 ts_tablet_manager.cc:1939] T 0d171635f93b403d929b3f18ca0d34f1 P c00fb34a6d504547b63fa36f866225d7: Deleting consensus metadata
I20260927 07:10:30.492304 14381 catalog_manager.cc:4841] TS 267dfef0c6224ae2af395d4eda085ddd (127.30.16.65:45537): tablet 0d171635f93b403d929b3f18ca0d34f1 (table test-table [id=94938fccc635407fb0a6a9ce903c4c34]) successfully deleted
I20260927 07:10:30.494011 14382 catalog_manager.cc:4841] TS c00fb34a6d504547b63fa36f866225d7 (127.30.16.67:32897): tablet 0d171635f93b403d929b3f18ca0d34f1 (table test-table [id=94938fccc635407fb0a6a9ce903c4c34]) successfully deleted
I20260927 07:10:30.494721 14884 ts_tablet_manager.cc:1918] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:10:30.494973 14884 log.cc:1196] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/0d171635f93b403d929b3f18ca0d34f1
I20260927 07:10:30.495621 14884 ts_tablet_manager.cc:1939] T 0d171635f93b403d929b3f18ca0d34f1 P fa9d9110b50449ec9f507a27a581c958: Deleting consensus metadata
I20260927 07:10:30.496876 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.460911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.30.16.65:45537)
0927 07:10:30.461136 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.461147 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.472833 (+ 11686us) server_negotiation.cc:407] Connection header received
0927 07:10:30.473878 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.473886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.473989 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.474077 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:30.479048 (+ 4971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.480237 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.482523 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.483252 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.483394 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.489111 (+ 5717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:30.489131 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:30.489151 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:30.489204 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:30.493298 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.494147 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.494159 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.494164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.494331 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.494892 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.494898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.494904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.495449 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:30.495822 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.496187 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.496447 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:10:30.497711 14577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:30.497788 14381 catalog_manager.cc:4841] TS fa9d9110b50449ec9f507a27a581c958 (127.30.16.66:39961): tablet 0d171635f93b403d929b3f18ca0d34f1 (table test-table [id=94938fccc635407fb0a6a9ce903c4c34]) successfully deleted
I20260927 07:10:30.619920 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14433
I20260927 07:10:30.644259 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14586
I20260927 07:10:30.660228 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.645101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52148 (local address 127.30.16.66:39961)
0927 07:10:30.645306 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.645312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.645336 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:10:30.645508 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.645515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.645603 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.645719 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:30.647125 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.648263 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.649729 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.650361 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.650495 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.653868 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:30.653884 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:30.653901 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:30.653943 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:30.656950 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.657894 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.657903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.657905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.658115 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.658544 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.658610 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.658617 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.659018 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:30.659313 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.659686 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.659859 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:10:30.660876 14709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:30.752244 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14586
I20260927 07:10:30.770282 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14717
I20260927 07:10:30.788280 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:30.771153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.30.16.67:32897)
0927 07:10:30.771376 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 07:10:30.771383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:10:30.771408 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:10:30.771590 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:10:30.771602 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:10:30.771697 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:10:30.771847 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:10:30.773287 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.774572 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.775976 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:10:30.776743 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:10:30.776886 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:10:30.779787 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:10:30.779803 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:10:30.779823 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:10:30.779875 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:10:30.784326 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.785107 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.785118 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.785121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.785408 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:10:30.785889 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:10:30.785894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:10:30.785898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:10:30.786420 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:10:30.786810 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0927 07:10:30.787504 (+ 694us) server_negotiation.cc:299] Negotiation successful
0927 07:10:30.787780 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260927 07:10:30.789193 14840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:30.912081 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14717
I20260927 07:10:30.932106 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14363
I20260927 07:10:30.933364 14426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:10:31.048506 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14363
I20260927 07:10:51.071764 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--webserver_interface=127.30.16.126
--webserver_port=38979
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:51.419690 14890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:51.420014 14890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:51.420149 14890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:51.427779 14890 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 07:10:51.427893 14890 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 07:10:51.427978 14890 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 07:10:51.428081 14890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:10:51.428160 14890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:51.428287 14890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:51.428357 14890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:10:51.428435 14890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:10:51.440239 14890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:51.441660 14890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:51.443209 14890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:51.455492 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:51.455509 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:51.455571 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:51.456367 14890 server_base.cc:1034] running on GCE node
I20260927 07:10:51.457194 14890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:51.458755 14890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:51.460002 14890 hybrid_clock.cc:648] HybridClock initialized: now 1790493051459944 us; error 40 us; skew 500 ppm
I20260927 07:10:51.460353 14890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:51 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493051, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:51.468954 14890 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:10:51.470726 14890 webserver.cc:460] Webserver started at http://127.30.16.126:38979/ using document root <none> and password file <none>
I20260927 07:10:51.471513 14890 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:51.471647 14890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:51.477421 14890 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 07:10:51.480638 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:51.482084 14890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:10:51.482290 14890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "5be94b8ece764074bd4abe5e970e814d"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "3fd7556322451c661933538e4207a431"
server_key_iv: "9580087c1e871de9cfe3356c818ff6ac"
server_key_version: "encryptionkey@0"
I20260927 07:10:51.483408 14890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:51.497965 14890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:51.502985 14890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:51.503284 14890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:51.522207 14890 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43019
I20260927 07:10:51.522243 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43019 every 8 connection(s)
I20260927 07:10:51.523856 14890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:10:51.527230 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14890
I20260927 07:10:51.528270 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:45537
--local_ip_for_outbound_sockets=127.30.16.65
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=45339
--webserver_interface=127.30.16.65
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:10:51.533026 14957 sys_catalog.cc:263] Verifying existing consensus state
I20260927 07:10:51.538328 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap starting.
I20260927 07:10:51.560351 14957 log.cc:824] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:10:51.576746 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 07:10:51.577416 14957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Bootstrap complete.
I20260927 07:10:51.586993 14957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:51.588162 14957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Initialized, Role: FOLLOWER
I20260927 07:10:51.588814 14957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:51.589071 14957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:10:51.589208 14957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:10:51.589471 14957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 2 FOLLOWER]: Advancing to term 3
I20260927 07:10:51.593667 14957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:51.594245 14957 leader_election.cc:302] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5be94b8ece764074bd4abe5e970e814d; no voters:
I20260927 07:10:51.595049 14957 leader_election.cc:288] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 07:10:51.595203 14961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 07:10:51.596344 14961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [term 3 LEADER]: Becoming Leader. State: Replica: 5be94b8ece764074bd4abe5e970e814d, State: Running, Role: LEADER
I20260927 07:10:51.597043 14961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } }
I20260927 07:10:51.597440 14957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:10:51.599685 14963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5be94b8ece764074bd4abe5e970e814d. Latest consensus state: current_term: 3 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:51.600075 14963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:51.602147 14962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5be94b8ece764074bd4abe5e970e814d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5be94b8ece764074bd4abe5e970e814d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43019 } } }
I20260927 07:10:51.602596 14962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d [sys.catalog]: This master's current role is: LEADER
I20260927 07:10:51.612027 14967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:10:51.624668 14967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:10:51.627682 14967 catalog_manager.cc:1259] Loaded cluster ID: abf825cee14240ea8b4208f34af1ee70
I20260927 07:10:51.628062 14967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:10:51.633755 14967 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:10:51.636153 14967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Loaded TSK: 0
I20260927 07:10:51.636461 14967 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5be94b8ece764074bd4abe5e970e814d: Loaded TSK: 1
F20260927 07:10:51.637985 14967 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1395e5e9ad google::LogMessage::Fail() at ??:0
@ 0x7f1395e6188a google::LogMessage::SendToLog() at ??:0
@ 0x7f1395e5e4de google::LogMessage::Flush() at ??:0
@ 0x7f1395e5fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1399bc27d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1399bb173e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f13ab88d1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f13ab88c783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f13ab9182a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f13ab9180c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f13ab9625e9 std::function<>::operator()() at ??:0
I20260927 07:10:51.709416 14981 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f13ab890cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f13ab88ec36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f13ab916610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f13ab916439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1398aaa345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1398aad91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1398aad799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1398a88eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f139bd626db start_thread at ??:0
@ 0x7f1394cf861f clone at ??:0
W20260927 07:10:51.959249 14959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:51.959595 14959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:51.959766 14959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:51.967365 14959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:51.967533 14959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:51.967724 14959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:10:51.977869 14959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:45537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=45339
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:51.979408 14959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:51.980906 14959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:51.995011 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:51.995081 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:51.995028 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:51.995949 14959 server_base.cc:1034] running on GCE node
I20260927 07:10:51.996546 14959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:51.997506 14959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:51.998728 14959 hybrid_clock.cc:648] HybridClock initialized: now 1790493051998686 us; error 44 us; skew 500 ppm
I20260927 07:10:51.999066 14959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:51 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493051, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:52.002306 14959 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:10:52.003953 14959 webserver.cc:460] Webserver started at http://127.30.16.65:45339/ using document root <none> and password file <none>
I20260927 07:10:52.004691 14959 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:52.004838 14959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:52.010532 14959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 07:10:52.013626 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.015327 14959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:52.015519 14959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "267dfef0c6224ae2af395d4eda085ddd"
format_stamp: "Formatted at 2026-09-27 07:10:05 on dist-test-slave-8vqf"
server_key: "0590c87ef2eb0ef1defad69f0205fd12"
server_key_iv: "8cceb54977e74f45f95c966fcad544bc"
server_key_version: "encryptionkey@0"
I20260927 07:10:52.016633 14959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:52.042754 14959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:52.047686 14959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:52.048003 14959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:52.049136 14959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:52.050860 14959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:52.050994 14959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.051143 14959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:52.051265 14959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.087839 14959 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:45537
I20260927 07:10:52.087932 15126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:45537 every 8 connection(s)
I20260927 07:10:52.089423 14959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:10:52.094725 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.092705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:37287)
0927 07:10:52.093543 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.093806 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":306,"threads_started":1}
W20260927 07:10:52.095453 15127 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43019 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43019: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
I20260927 07:10:52.096258 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.095933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:37045)
0927 07:10:52.096093 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.096158 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 07:10:52.097078 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.096765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:37003)
0927 07:10:52.096900 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.096974 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:10:52.097307 15127 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:10:52.097348 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 14959
I20260927 07:10:52.098049 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:39961
--local_ip_for_outbound_sockets=127.30.16.66
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=45769
--webserver_interface=127.30.16.66
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:10:52.441473 15130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:52.441789 15130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:52.441959 15130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:52.449462 15130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:52.449592 15130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:52.449800 15130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:10:52.459991 15130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:39961
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=45769
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:52.461417 15130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:52.462862 15130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:52.476827 15136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:52.476843 15135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:52.476855 15138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:52.483158 15130 server_base.cc:1034] running on GCE node
I20260927 07:10:52.483917 15130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:52.485016 15130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:52.486297 15130 hybrid_clock.cc:648] HybridClock initialized: now 1790493052486217 us; error 49 us; skew 500 ppm
I20260927 07:10:52.486711 15130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:52 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493052, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:52.490044 15130 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:10:52.491829 15130 webserver.cc:460] Webserver started at http://127.30.16.66:45769/ using document root <none> and password file <none>
I20260927 07:10:52.492594 15130 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:52.492748 15130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:52.498466 15130 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 07:10:52.501467 15145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.503151 15130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 07:10:52.503364 15130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "fa9d9110b50449ec9f507a27a581c958"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "acd632b9fc5b46884755ce0fb6dea4f9"
server_key_iv: "4bc1f8c51154bf8e5f712281f7c2b431"
server_key_version: "encryptionkey@0"
I20260927 07:10:52.504578 15130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:52.522503 15130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:52.527565 15130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:52.527894 15130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:52.529030 15130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:52.530512 15130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:52.530687 15130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 07:10:52.530833 15130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:52.530932 15130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.569940 15130 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:39961
I20260927 07:10:52.570030 15257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:39961 every 8 connection(s)
I20260927 07:10:52.571486 15130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:10:52.573642 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15130
I20260927 07:10:52.574292 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:32897
--local_ip_for_outbound_sockets=127.30.16.67
--tserver_master_addrs=127.30.16.126:43019
--webserver_port=34355
--webserver_interface=127.30.16.67
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:10:52.577353 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.574868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:43737)
0927 07:10:52.575866 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.576153 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":391,"threads_started":1}
W20260927 07:10:52.578174 15258 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43019 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43019: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
I20260927 07:10:52.579267 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.578814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:51431)
0927 07:10:52.579009 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.579107 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 07:10:52.580315 15260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:52.579890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:54161)
0927 07:10:52.580059 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:52.580163 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 07:10:52.580615 15258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 07:10:52.917224 15261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:10:52.917521 15261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:10:52.917687 15261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:10:52.925403 15261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:10:52.925530 15261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:10:52.925765 15261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:10:52.935865 15261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:32897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=34355
--tserver_master_addrs=127.30.16.126:43019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:10:52.937256 15261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:10:52.938628 15261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:10:52.952551 15266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:52.952489 15267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:10:52.952603 15269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:10:52.956192 15261 server_base.cc:1034] running on GCE node
I20260927 07:10:52.956847 15261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:10:52.957819 15261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:10:52.959015 15261 hybrid_clock.cc:648] HybridClock initialized: now 1790493052958976 us; error 36 us; skew 500 ppm
I20260927 07:10:52.959311 15261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:10:52 dist-test-slave-8vqf krb5kdc[13811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493052, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:10:52.962438 15261 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:10:52.964018 15261 webserver.cc:460] Webserver started at http://127.30.16.67:34355/ using document root <none> and password file <none>
I20260927 07:10:52.964725 15261 fs_manager.cc:359] Metadata directory not provided
I20260927 07:10:52.964828 15261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:10:52.970362 15261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 07:10:52.973165 15276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:52.974640 15261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:10:52.974814 15261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "c00fb34a6d504547b63fa36f866225d7"
format_stamp: "Formatted at 2026-09-27 07:10:06 on dist-test-slave-8vqf"
server_key: "8b790937faabf3bc7e6f82f402eb4d88"
server_key_iv: "98d90f153792c0f9c7c2d67f46cd08cc"
server_key_version: "encryptionkey@0"
I20260927 07:10:52.975821 15261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:10:53.002247 15261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:10:53.007234 15261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:10:53.007544 15261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:10:53.008769 15261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:10:53.010424 15261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:10:53.010516 15261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:53.010701 15261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:10:53.010783 15261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:10:53.046495 15261 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:32897
I20260927 07:10:53.046643 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:32897 every 8 connection(s)
I20260927 07:10:53.048112 15261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:10:53.050935 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15261
I20260927 07:10:53.054029 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:53.051757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:42997)
0927 07:10:53.052616 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:53.053070 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":350,"threads_started":1}
W20260927 07:10:53.054704 15389 heartbeater.cc:644] Failed to heartbeat to 127.30.16.126:43019 (0 consecutive failures): Network error: Failed to ping master at 127.30.16.126:43019: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
I20260927 07:10:53.055576 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:53.055228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:35979)
0927 07:10:53.055389 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:53.055466 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 07:10:53.056449 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:53.056090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:55129)
0927 07:10:53.056228 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:53.056324 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 07:10:53.056672 15389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 07:10:53.098766 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:53.097879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:47489)
0927 07:10:53.098363 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:53.098452 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260927 07:10:53.582080 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:53.581235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:46355)
0927 07:10:53.581727 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:53.581854 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":226,"threads_started":1}
I20260927 07:10:54.058128 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:54.057257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:55985)
0927 07:10:54.057792 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:54.057887 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":253,"threads_started":1}
I20260927 07:10:54.100695 15396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:54.099831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:57979)
0927 07:10:54.100441 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:54.100536 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":318,"threads_started":1}
I20260927 07:10:54.583905 15397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:54.583145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:47287)
0927 07:10:54.583634 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:54.583719 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260927 07:10:55.059916 15398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:55.059179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:52879)
0927 07:10:55.059658 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:55.059746 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":236,"threads_started":1}
I20260927 07:10:55.102471 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:55.101601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:32811)
0927 07:10:55.102144 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:55.102279 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":242,"threads_started":1}
I20260927 07:10:55.585672 15400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:55.584933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:37457)
0927 07:10:55.585412 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:55.585497 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260927 07:10:56.061702 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:56.060953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:47051)
0927 07:10:56.061430 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:56.061519 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":250,"threads_started":1}
I20260927 07:10:56.104799 15402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:56.103738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:38889)
0927 07:10:56.104392 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:56.104520 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":316,"threads_started":1}
I20260927 07:10:56.587388 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:56.586676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:38697)
0927 07:10:56.587121 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:56.587203 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260927 07:10:57.063529 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:57.062748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:39917)
0927 07:10:57.063263 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:57.063353 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260927 07:10:57.106734 15405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:57.105854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:57151)
0927 07:10:57.106368 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:57.106464 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
I20260927 07:10:57.589192 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:57.588377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:37335)
0927 07:10:57.588941 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:57.589020 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":314,"threads_started":1}
I20260927 07:10:58.065238 15407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:58.064500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:45157)
0927 07:10:58.064986 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:58.065070 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":257,"threads_started":1}
I20260927 07:10:58.108345 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:58.107606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:45695)
0927 07:10:58.108061 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:58.108154 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260927 07:10:58.591066 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:58.590065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:41017)
0927 07:10:58.590670 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:58.590839 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":327,"threads_started":1}
I20260927 07:10:59.067135 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:59.066263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:56409)
0927 07:10:59.066854 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:59.066957 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":333,"threads_started":1}
I20260927 07:10:59.110316 15411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:59.109346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:50245)
0927 07:10:59.109965 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:59.110057 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":319,"threads_started":1}
I20260927 07:10:59.592767 15412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:10:59.592009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:54489)
0927 07:10:59.592479 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 07:10:59.592557 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":230,"threads_started":1}
I20260927 07:11:00.069031 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:00.068214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:37727)
0927 07:11:00.068716 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:00.068822 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":213,"threads_started":1}
I20260927 07:11:00.112052 15414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:00.111298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:45123)
0927 07:11:00.111770 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:00.111885 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":218,"threads_started":1}
I20260927 07:11:00.594409 15415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:00.593703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:43571)
0927 07:11:00.594158 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:00.594241 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260927 07:11:01.070820 15416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:01.070040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:58533)
0927 07:11:01.070499 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:01.070647 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260927 07:11:01.113790 15417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:01.113028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:59605)
0927 07:11:01.113521 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:01.113609 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
I20260927 07:11:01.596161 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:01.595393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:54757)
0927 07:11:01.595897 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:01.595986 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":243,"threads_started":1}
I20260927 07:11:02.072530 15419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:02.071738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:36809)
0927 07:11:02.072246 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:02.072355 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":239,"threads_started":1}
I20260927 07:11:02.115803 15420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:02.115075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:48051)
0927 07:11:02.115542 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:02.115629 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
I20260927 07:11:02.597877 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:02.597048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:59913)
0927 07:11:02.597622 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:02.597708 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":337,"threads_started":1}
I20260927 07:11:03.074352 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:03.073559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:57073)
0927 07:11:03.074070 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:03.074160 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":288,"threads_started":1}
I20260927 07:11:03.117581 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:03.116833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:57407)
0927 07:11:03.117295 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:03.117388 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260927 07:11:03.599622 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:03.598930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:49129)
0927 07:11:03.599346 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:03.599435 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260927 07:11:04.076145 15425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:04.075374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:35723)
0927 07:11:04.075863 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:04.075966 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260927 07:11:04.119441 15426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:04.118602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:43353)
0927 07:11:04.119183 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:04.119269 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":335,"threads_started":1}
I20260927 07:11:04.601478 15427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:04.600740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:45485)
0927 07:11:04.601214 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:04.601320 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
I20260927 07:11:05.078372 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:05.077326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:55649)
0927 07:11:05.077987 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:05.078108 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":357,"threads_started":1}
I20260927 07:11:05.121429 15429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:05.120563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:38517)
0927 07:11:05.121110 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:05.121239 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":245,"threads_started":1}
I20260927 07:11:05.603825 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:05.602707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:51759)
0927 07:11:05.603376 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:05.603553 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":388,"threads_started":1}
I20260927 07:11:06.080430 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:06.079596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:55849)
0927 07:11:06.080137 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:06.080238 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":268,"threads_started":1}
I20260927 07:11:06.123416 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:06.122325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:44553)
0927 07:11:06.122991 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:06.123123 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":300,"threads_started":1}
I20260927 07:11:06.605441 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:06.604712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:48173)
0927 07:11:06.605165 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:06.605258 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260927 07:11:07.082340 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:07.081584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:57187)
0927 07:11:07.082074 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:07.082158 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":263,"threads_started":1}
I20260927 07:11:07.125488 15435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:07.124747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:54151)
0927 07:11:07.125192 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:07.125303 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":218,"threads_started":1}
I20260927 07:11:07.607585 15436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:07.606854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.66:36655)
0927 07:11:07.607309 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:07.607402 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":234,"threads_started":1}
I20260927 07:11:08.053562 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 14959
I20260927 07:11:08.072062 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:08.054897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.30.16.65:45537)
0927 07:11:08.055518 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:08.055532 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:08.055574 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:11:08.057140 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:08.057150 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:08.057603 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:08.057915 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:08.059443 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.060723 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.062102 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.062897 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.063038 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:08.065696 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:08.065704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:08.065720 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:08.065759 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:08.068748 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.069523 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.069529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.069535 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.069697 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.070070 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.070073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.070075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.070675 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:08.070940 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:08.071296 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 07:11:08.071743 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260927 07:11:08.073726 15122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:08.084376 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:08.083596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.67:32881)
0927 07:11:08.084071 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:08.084179 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":253,"threads_started":1}
I20260927 07:11:08.163404 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:08.162312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43019 (local address 127.30.16.65:43967)
0927 07:11:08.162968 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:08.163141 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.126:43019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":347,"threads_started":1}
I20260927 07:11:08.167095 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 14959
I20260927 07:11:08.181973 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15130
I20260927 07:11:08.200919 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:08.183061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.30.16.66:39961)
0927 07:11:08.183606 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:08.183620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:08.183658 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:11:08.185242 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:08.185252 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:08.185711 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:08.186082 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:08.187734 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.189011 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.190338 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.191091 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.191291 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:08.194068 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:08.194077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:08.194094 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:08.194140 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:08.197095 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.197793 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.197800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.197809 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.197983 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.198388 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.198394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.198397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.199318 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:08.199692 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:08.200168 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 07:11:08.200609 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
I20260927 07:11:08.202433 15253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:08.290622 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15130
I20260927 07:11:08.310278 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15261
I20260927 07:11:08.329226 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:08.311351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32980 (local address 127.30.16.67:32897)
0927 07:11:08.311961 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:08.311973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:08.312019 (+ 46us) server_negotiation.cc:407] Connection header received
0927 07:11:08.313695 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:08.313705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:08.314167 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:08.314494 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:08.316078 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.317379 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.318842 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:08.319564 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:08.319703 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:08.322471 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:08.322479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:08.322497 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:08.322540 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:08.325670 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.326395 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.326402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.326410 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.326725 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:08.327103 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:08.327106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:08.327108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:08.327678 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:08.327923 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:08.328454 (+ 531us) server_negotiation.cc:299] Negotiation successful
0927 07:11:08.328861 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":290,"threads_started":1}
I20260927 07:11:08.330878 15384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:08.415992 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15261
2026-09-27T07:11:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65005 ms)
[----------] 30 tests from SecurityITest (300423 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:08 dist-test-slave-8vqf krb5kdc[15447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:08 dist-test-slave-8vqf krb5kdc[15447](info): set up 2 sockets
Sep 27 07:11:08 dist-test-slave-8vqf krb5kdc[15447](info): commencing operation
krb5kdc: starting...
W20260927 07:11:09.574909 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.118s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:09 dist-test-slave-8vqf krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:11:09.836062 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42645
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:42645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:10.190243 15460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:10.190709 15460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:10.190899 15460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:10.198907 15460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:10.199087 15460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:10.199186 15460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:10.199302 15460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:10.199390 15460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:10.210345 15460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:42645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:42645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:10.211937 15460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:10.213352 15460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:10.226912 15465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:10.226854 15468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:10.226866 15466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:10.227241 15460 server_base.cc:1034] running on GCE node
I20260927 07:11:10.228416 15460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:10.229967 15460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:10.231205 15460 hybrid_clock.cc:648] HybridClock initialized: now 1790493070231136 us; error 68 us; skew 500 ppm
I20260927 07:11:10.231580 15460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:10 dist-test-slave-8vqf krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:10.234903 15460 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:11:10.236380 15460 webserver.cc:460] Webserver started at http://127.30.16.126:36635/ using document root <none> and password file <none>
I20260927 07:11:10.237052 15460 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:10.237152 15460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:10.237406 15460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:10.240613 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "2473316ce9e3459097659bf8ef7130bd"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "b6fbf0ba1c0e9e43cf8651783177be68"
server_key_iv: "12e76667cba80e37e60856d24d5aa3ca"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.241328 15460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "2473316ce9e3459097659bf8ef7130bd"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "b6fbf0ba1c0e9e43cf8651783177be68"
server_key_iv: "12e76667cba80e37e60856d24d5aa3ca"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.246690 15460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260927 07:11:10.250270 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:10.251767 15460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:11:10.251933 15460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "2473316ce9e3459097659bf8ef7130bd"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "b6fbf0ba1c0e9e43cf8651783177be68"
server_key_iv: "12e76667cba80e37e60856d24d5aa3ca"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.252103 15460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:10.274240 15460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:10.278995 15460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:10.279238 15460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:10.298228 15460 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:42645
I20260927 07:11:10.298255 15526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:42645 every 8 connection(s)
I20260927 07:11:10.299800 15460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:10.301890 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15460
I20260927 07:11:10.302120 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:10.302868 30785 external_mini_cluster.cc:1428] Setting key 9cd1da903624b469e5ac7b521b5d9442
I20260927 07:11:10.304457 15527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:11:10 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:10.320765 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: Bootstrap starting.
I20260927 07:11:10.324551 15527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:10.325680 15527 log.cc:824] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:10.329030 15527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: No bootstrap required, opened a new log
I20260927 07:11:10.329363 15530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:10.304943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55700 (local address 127.30.16.126:42645)
0927 07:11:10.305789 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:10.305814 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:10.305869 (+ 55us) server_negotiation.cc:407] Connection header received
0927 07:11:10.308458 (+ 2589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:10.308491 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:10.309079 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:10.309581 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:10.311879 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:10.313295 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:10.315281 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:10.316035 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:10.316179 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:10.321262 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:10.321272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:10.321291 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:10.321337 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:10.324572 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:10.325389 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:10.325396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:10.325407 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:10.325594 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:10.326085 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:10.326090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:10.326092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:10.326759 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:10.327110 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:10.327597 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 07:11:10.328132 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":363,"threads_started":1}
I20260927 07:11:10.337595 15527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } }
I20260927 07:11:10.338054 15527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:10.338205 15527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2473316ce9e3459097659bf8ef7130bd, State: Initialized, Role: FOLLOWER
I20260927 07:11:10.338984 15527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } }
I20260927 07:11:10.339270 15527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:10.339421 15527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:10.339627 15527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:10.341867 15527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } }
I20260927 07:11:10.342423 15527 leader_election.cc:302] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2473316ce9e3459097659bf8ef7130bd; no voters:
I20260927 07:11:10.343227 15527 leader_election.cc:288] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:10.343361 15532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:10.344740 15532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [term 1 LEADER]: Becoming Leader. State: Replica: 2473316ce9e3459097659bf8ef7130bd, State: Running, Role: LEADER
I20260927 07:11:10.345278 15527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:10.345379 15532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } }
I20260927 07:11:10.348462 15534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2473316ce9e3459097659bf8ef7130bd. Latest consensus state: current_term: 1 leader_uuid: "2473316ce9e3459097659bf8ef7130bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } } }
I20260927 07:11:10.348883 15534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:10.349427 15533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2473316ce9e3459097659bf8ef7130bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2473316ce9e3459097659bf8ef7130bd" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 42645 } } }
I20260927 07:11:10.349841 15533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:10.353231 15542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:10.358608 15542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:10.368235 15542 catalog_manager.cc:1347] Generated new cluster ID: ff08064339fb47e3be00737e8870756d
I20260927 07:11:10.368391 15542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:10.391844 15542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:10.393262 15542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:10.410759 15542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: Generated new TSK 0
I20260927 07:11:10.411765 15542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:11:10.487083 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:10.860387 15553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:10.860718 15553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:10.860880 15553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:10.868741 15553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:10.868878 15553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:10.869057 15553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:10.879555 15553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:10.881006 15553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:10.882465 15553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:10.896926 15558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:10.896973 15559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:10.896891 15561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:10.897970 15553 server_base.cc:1034] running on GCE node
I20260927 07:11:10.898741 15553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:10.899811 15553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:10.901044 15553 hybrid_clock.cc:648] HybridClock initialized: now 1790493070900951 us; error 93 us; skew 500 ppm
I20260927 07:11:10.901387 15553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:10 dist-test-slave-8vqf krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:10.904969 15553 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:11:10.906687 15553 webserver.cc:460] Webserver started at http://127.30.16.65:34849/ using document root <none> and password file <none>
I20260927 07:11:10.907464 15553 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:10.907622 15553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:10.907961 15553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:10.911425 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "60bc382cef4041ef9623bf0a18bb5d27"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "aac7054770e13b610e849de2f6185ff7"
server_key_iv: "22969cbec24ca6543b27a9e58503d5eb"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.912235 15553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "60bc382cef4041ef9623bf0a18bb5d27"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "aac7054770e13b610e849de2f6185ff7"
server_key_iv: "22969cbec24ca6543b27a9e58503d5eb"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.917912 15553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:11:10.921819 15568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:10.923473 15553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:10.923686 15553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "60bc382cef4041ef9623bf0a18bb5d27"
format_stamp: "Formatted at 2026-09-27 07:11:10 on dist-test-slave-8vqf"
server_key: "aac7054770e13b610e849de2f6185ff7"
server_key_iv: "22969cbec24ca6543b27a9e58503d5eb"
server_key_version: "encryptionkey@0"
I20260927 07:11:10.923913 15553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:10.954784 15553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:10.959662 15553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:10.959992 15553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:10.961055 15553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:10.962491 15553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:10.962617 15553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 07:11:10.962774 15553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:10.962881 15553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:10.996951 15553 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:43157
I20260927 07:11:10.997056 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:43157 every 8 connection(s)
I20260927 07:11:10.998327 15553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:11.007364 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15553
I20260927 07:11:11.007603 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:11.008240 30785 external_mini_cluster.cc:1428] Setting key 80ed2f6d5acb114b24aeb7c8dc3275dd
Sep 27 07:11:11 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493070, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:11.020700 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:11.001049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:58043 (local address 127.30.16.126:42645)
0927 07:11:11.001497 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:11.001504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:11.002684 (+ 1180us) server_negotiation.cc:407] Connection header received
0927 07:11:11.004658 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:11.004664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:11.004750 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:11.004915 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:11.006497 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.007295 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.009272 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.009904 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.010043 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:11.014541 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:11.014592 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:11.014597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:11.014644 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:11.017355 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:11.018144 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:11.018150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:11.018153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:11.018290 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:11.018842 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:11.018853 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:11.018855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:11.019141 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:11.019307 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:11.020068 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 07:11:11.020304 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":204,"threads_started":1}
I20260927 07:11:11.021847 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:11.001728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42645 (local address 127.30.16.65:58043)
0927 07:11:11.002454 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:11.002495 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:11.004344 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:11.005123 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:11.005132 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:11.005694 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:11.006319 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:11.006337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.007497 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.007501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:11.009105 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:11.009123 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.010193 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.010202 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:11.010513 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:11.011329 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:11.011362 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:11.014250 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:11.017524 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:11.017534 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:11.017554 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:11.018010 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:11.018411 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:11.018416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:11.018420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:11.018713 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:11.019279 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:11.019288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:11.019687 (+ 399us) client_negotiation.cc:769] Sending connection context
0927 07:11:11.019974 (+ 287us) client_negotiation.cc:240] Negotiation successful
0927 07:11:11.020572 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":292,"threads_started":1}
I20260927 07:11:11.023689 15681 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42645
I20260927 07:11:11.024024 15681 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:11.024794 15681 heartbeater.cc:511] Master 127.30.16.126:42645 requested a full tablet report, sending...
I20260927 07:11:11.027227 15492 ts_manager.cc:194] Registered new tserver with Master: 60bc382cef4041ef9623bf0a18bb5d27 (127.30.16.65:43157)
I20260927 07:11:11.029506 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:58043
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:11:11.067103 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:11.414160 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:11.414471 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:11.414656 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:11.422427 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:11.422611 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:11.422806 15687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:11.433151 15687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:11.434597 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:11.436014 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:11.450433 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:11.450538 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:11.450492 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:11.451634 15687 server_base.cc:1034] running on GCE node
I20260927 07:11:11.452384 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:11.453377 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:11.454666 15687 hybrid_clock.cc:648] HybridClock initialized: now 1790493071454618 us; error 41 us; skew 500 ppm
I20260927 07:11:11.454996 15687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:11 dist-test-slave-8vqf krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:11.467769 15687 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:11:11.469578 15687 webserver.cc:460] Webserver started at http://127.30.16.66:37189/ using document root <none> and password file <none>
I20260927 07:11:11.470362 15687 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:11.470520 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:11.470911 15687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:11.474025 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "ec956958d9754431af14d29ec76e0b52"
format_stamp: "Formatted at 2026-09-27 07:11:11 on dist-test-slave-8vqf"
server_key: "8b4d86fd4e280ed683c52152d6d37193"
server_key_iv: "0e5ef0982edfe83c51f0253dbbac1e19"
server_key_version: "encryptionkey@0"
I20260927 07:11:11.474972 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "ec956958d9754431af14d29ec76e0b52"
format_stamp: "Formatted at 2026-09-27 07:11:11 on dist-test-slave-8vqf"
server_key: "8b4d86fd4e280ed683c52152d6d37193"
server_key_iv: "0e5ef0982edfe83c51f0253dbbac1e19"
server_key_version: "encryptionkey@0"
I20260927 07:11:11.480662 15687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:11.484690 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:11.486304 15687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:11:11.486524 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "ec956958d9754431af14d29ec76e0b52"
format_stamp: "Formatted at 2026-09-27 07:11:11 on dist-test-slave-8vqf"
server_key: "8b4d86fd4e280ed683c52152d6d37193"
server_key_iv: "0e5ef0982edfe83c51f0253dbbac1e19"
server_key_version: "encryptionkey@0"
I20260927 07:11:11.486784 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:11.509671 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:11.514163 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:11.514506 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:11.515662 15687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:11.517148 15687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:11.517262 15687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:11.517431 15687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:11.517508 15687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:11.553814 15687 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:45697
I20260927 07:11:11.553870 15814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:45697 every 8 connection(s)
I20260927 07:11:11.555296 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:11.562808 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15687
I20260927 07:11:11.563027 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:11.563585 30785 external_mini_cluster.cc:1428] Setting key a167acd7640224fca9ef0b78fcf95bb9
Sep 27 07:11:11 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493071, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:11.579193 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:11.558283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:37311 (local address 127.30.16.126:42645)
0927 07:11:11.558798 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:11.558803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:11.559767 (+ 964us) server_negotiation.cc:407] Connection header received
0927 07:11:11.561704 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:11.561709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:11.561809 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:11.561937 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:11.564296 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.565149 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.567602 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.568053 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.568161 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:11.572874 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:11.572879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:11.572883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:11.572918 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:11.575600 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:11.576456 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:11.576464 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:11.576471 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:11.576630 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:11.577075 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:11.577083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:11.577089 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:11.577490 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:11.577642 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:11.578377 (+ 735us) server_negotiation.cc:299] Negotiation successful
0927 07:11:11.578675 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":298,"threads_started":1}
I20260927 07:11:11.580057 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:11.558850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42645 (local address 127.30.16.66:37311)
0927 07:11:11.559594 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:11.559635 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:11.561395 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:11.562211 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:11.562221 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:11.563152 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:11.564040 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:11.564068 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.565335 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.565343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:11.567424 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:11.567444 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:11.568291 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:11.568301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:11.568664 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:11.569464 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:11.569500 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:11.572579 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:11.575764 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:11.575777 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:11.575800 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:11.576311 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:11.576765 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:11.576768 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:11.576771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:11.576957 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:11.577618 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:11.577624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:11.577979 (+ 355us) client_negotiation.cc:769] Sending connection context
0927 07:11:11.578219 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 07:11:11.578818 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":283,"threads_started":1}
I20260927 07:11:11.581600 15815 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42645
I20260927 07:11:11.581919 15815 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:11.582743 15815 heartbeater.cc:511] Master 127.30.16.126:42645 requested a full tablet report, sending...
I20260927 07:11:11.584329 15492 ts_manager.cc:194] Registered new tserver with Master: ec956958d9754431af14d29ec76e0b52 (127.30.16.66:45697)
I20260927 07:11:11.585158 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:37311
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:11:11.621518 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:34925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:11.968678 15821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:11.969029 15821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:11.969226 15821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:11.976766 15821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:11.976933 15821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:11.977108 15821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:11.987274 15821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:34925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:42645
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:11.988642 15821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:11.989995 15821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:12.005950 15826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:12.005976 15827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:12.005947 15829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:12.006907 15821 server_base.cc:1034] running on GCE node
I20260927 07:11:12.007644 15821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:12.008692 15821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:12.009939 15821 hybrid_clock.cc:648] HybridClock initialized: now 1790493072009884 us; error 51 us; skew 500 ppm
I20260927 07:11:12.010284 15821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:12 dist-test-slave-8vqf krb5kdc[15447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493072, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:12.013612 15821 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:11:12.015239 15821 webserver.cc:460] Webserver started at http://127.30.16.67:34411/ using document root <none> and password file <none>
I20260927 07:11:12.015993 15821 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:12.016144 15821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:12.016484 15821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:12.019624 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d"
format_stamp: "Formatted at 2026-09-27 07:11:12 on dist-test-slave-8vqf"
server_key: "c37aad10822aa7110506e6ddc7452a97"
server_key_iv: "ef9525265f42722895e58c0a6b1d35f5"
server_key_version: "encryptionkey@0"
I20260927 07:11:12.020437 15821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d"
format_stamp: "Formatted at 2026-09-27 07:11:12 on dist-test-slave-8vqf"
server_key: "c37aad10822aa7110506e6ddc7452a97"
server_key_iv: "ef9525265f42722895e58c0a6b1d35f5"
server_key_version: "encryptionkey@0"
I20260927 07:11:12.025858 15821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 07:11:12.029692 15836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:12.031211 15821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:11:12.031409 15821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d"
format_stamp: "Formatted at 2026-09-27 07:11:12 on dist-test-slave-8vqf"
server_key: "c37aad10822aa7110506e6ddc7452a97"
server_key_iv: "ef9525265f42722895e58c0a6b1d35f5"
server_key_version: "encryptionkey@0"
I20260927 07:11:12.031635 15821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:12.033041 15681 heartbeater.cc:503] Master 127.30.16.126:42645 was elected leader, sending a full tablet report...
I20260927 07:11:12.044567 15821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:12.048985 15821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:12.049299 15821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:12.050282 15821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:12.051788 15821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:12.051867 15821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:12.052001 15821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:12.052052 15821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:12.087546 15821 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:36935
I20260927 07:11:12.087564 15948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:36935 every 8 connection(s)
I20260927 07:11:12.088937 15821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:12.097826 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 15821
I20260927 07:11:12.098055 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:12.098513 30785 external_mini_cluster.cc:1428] Setting key e950873aa8008d3b2f2cccf7ed6f00bd
Sep 27 07:11:12 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493072, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:12.110018 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.091739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:45621 (local address 127.30.16.126:42645)
0927 07:11:12.092217 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.092222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.093387 (+ 1165us) server_negotiation.cc:407] Connection header received
0927 07:11:12.095388 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.095394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.095489 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.095609 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:12.097194 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.098010 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.100081 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.100715 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.100841 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.104606 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:12.104611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:12.104615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:12.104650 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:12.107234 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.107979 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.107983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.107985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.108109 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.108493 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.108496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.108498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.108772 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:12.108881 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.109538 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.109713 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260927 07:11:12.111033 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.092365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:42645 (local address 127.30.16.67:45621)
0927 07:11:12.093170 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.093259 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.095086 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.095870 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.095879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.096426 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:12.097017 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.097034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.098171 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.098183 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.099900 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.099922 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.100967 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.100974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.101286 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.102093 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:12.102116 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:12.104403 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:12.107380 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:12.107387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:12.107402 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:12.107833 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:12.108199 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:12.108202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:12.108205 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:12.108388 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:12.108876 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:12.108882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:12.109206 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.109431 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.109841 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260927 07:11:12.112694 15949 heartbeater.cc:347] Connected to a master server at 127.30.16.126:42645
I20260927 07:11:12.113047 15949 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:12.113817 15949 heartbeater.cc:511] Master 127.30.16.126:42645 requested a full tablet report, sending...
I20260927 07:11:12.115268 15492 ts_manager.cc:194] Registered new tserver with Master: dc44e3dff1db4d8e9065fdfd77eacb0d (127.30.16.67:36935)
I20260927 07:11:12.116055 15492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:45621
I20260927 07:11:12.127259 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:12.144825 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.132008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.30.16.126:42645)
0927 07:11:12.132215 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.132220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.132656 (+ 436us) server_negotiation.cc:407] Connection header received
0927 07:11:12.132802 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.132812 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.132879 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.132966 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:12.134430 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.135247 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.136709 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.137262 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.137365 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.139781 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:12.139786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:12.139789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:12.139840 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:12.142271 (+ 2431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.143025 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.143029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.143031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.143182 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.143530 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.143533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.143535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.143887 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:12.144072 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.144380 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.144565 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:11:12.149283 15492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:12.156538 15492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:12.181324 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.170153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.30.16.65:43157)
0927 07:11:12.170804 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.170817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.172338 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 07:11:12.172691 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.172703 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.173078 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.173359 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:12.174611 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.176259 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.178645 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.180093 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.180259 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.180523 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.180801 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.180984 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":344,"threads_started":1}
I20260927 07:11:12.181478 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.169628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:43157 (local address 127.0.0.1:55638)
0927 07:11:12.172187 (+ 2559us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.172208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.172467 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.173771 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.173782 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.173840 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:12.174298 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.174310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.176370 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.176376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.178444 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.178456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.180332 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.180337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.180520 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.180581 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.180742 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.180958 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2391,"mutex_wait_us":109,"thread_start_us":2194,"threads_started":1}
I20260927 07:11:12.182235 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.169612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.30.16.66:45697)
0927 07:11:12.170286 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.170293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.173000 (+ 2707us) server_negotiation.cc:407] Connection header received
0927 07:11:12.173138 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.173148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.173476 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.173711 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:12.175230 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.176641 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.179313 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.180804 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.180944 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.181148 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.181715 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.181905 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":381,"threads_started":1}
I20260927 07:11:12.182978 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.169571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:45697 (local address 127.0.0.1:43210)
0927 07:11:12.171402 (+ 1831us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.171442 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.171646 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.174425 (+ 2779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.174429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.174460 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:12.175009 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.175025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.176779 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.176785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.179116 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.179133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.181300 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.181308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.181505 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.181539 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.182439 (+ 900us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.182600 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1578,"thread_start_us":1434,"threads_started":1}
I20260927 07:11:12.184504 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.171118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.30.16.67:36935)
0927 07:11:12.174571 (+ 3453us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.174583 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.175226 (+ 643us) server_negotiation.cc:407] Connection header received
0927 07:11:12.175536 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.175544 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.175789 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.175963 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:12.177096 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.178758 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.181241 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.182727 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.182865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.183113 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.184042 (+ 929us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.184227 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3210,"thread_start_us":3016,"threads_started":1}
I20260927 07:11:12.185254 15967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.171663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36935 (local address 127.0.0.1:45706)
0927 07:11:12.175111 (+ 3448us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.175131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.175347 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.176300 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.176307 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.176333 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:12.176888 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.176907 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.178919 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.178925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.181062 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.181076 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.183647 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.183654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.183830 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.183861 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.184710 (+ 849us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.184877 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3331,"mutex_wait_us":20,"thread_start_us":3158,"threads_started":1}
I20260927 07:11:12.186151 15616 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c (DEFAULT_TABLE table=test-table [id=4afd87485f0d4d80bb53de97bc27b72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:12.187340 15750 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c (DEFAULT_TABLE table=test-table [id=4afd87485f0d4d80bb53de97bc27b72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:12.187863 15616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ca6528ce1c406b9a4823b7d0c70d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:12.188549 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ca6528ce1c406b9a4823b7d0c70d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:12.189067 15884 tablet_service.cc:1467] Processing CreateTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c (DEFAULT_TABLE table=test-table [id=4afd87485f0d4d80bb53de97bc27b72e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:12.190459 15884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ca6528ce1c406b9a4823b7d0c70d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:12.200428 15968 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Bootstrap starting.
I20260927 07:11:12.203302 15969 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Bootstrap starting.
I20260927 07:11:12.205217 15968 tablet_bootstrap.cc:654] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:12.206194 15970 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Bootstrap starting.
I20260927 07:11:12.206621 15968 log.cc:824] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:12.207131 15969 tablet_bootstrap.cc:654] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:12.209164 15969 log.cc:824] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:12.209679 15970 tablet_bootstrap.cc:654] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:12.210933 15970 log.cc:824] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:12.213923 15970 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: No bootstrap required, opened a new log
I20260927 07:11:12.214349 15970 ts_tablet_manager.cc:1397] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 07:11:12.216168 15968 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: No bootstrap required, opened a new log
I20260927 07:11:12.216485 15968 ts_tablet_manager.cc:1397] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260927 07:11:12.216611 15969 tablet_bootstrap.cc:492] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: No bootstrap required, opened a new log
I20260927 07:11:12.216904 15969 ts_tablet_manager.cc:1397] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260927 07:11:12.224566 15968 raft_consensus.cc:348] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.225010 15968 raft_consensus.cc:374] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:12.225134 15968 raft_consensus.cc:729] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec956958d9754431af14d29ec76e0b52, State: Initialized, Role: FOLLOWER
I20260927 07:11:12.225791 15968 consensus_queue.cc:260] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.225692 15970 raft_consensus.cc:348] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.226179 15970 raft_consensus.cc:374] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:12.226339 15970 raft_consensus.cc:729] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc44e3dff1db4d8e9065fdfd77eacb0d, State: Initialized, Role: FOLLOWER
I20260927 07:11:12.226987 15815 heartbeater.cc:503] Master 127.30.16.126:42645 was elected leader, sending a full tablet report...
I20260927 07:11:12.227183 15970 consensus_queue.cc:260] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.227782 15968 ts_tablet_manager.cc:1428] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260927 07:11:12.228041 15969 raft_consensus.cc:348] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.228502 15969 raft_consensus.cc:374] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:12.228693 15969 raft_consensus.cc:729] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bc382cef4041ef9623bf0a18bb5d27, State: Initialized, Role: FOLLOWER
I20260927 07:11:12.228796 15949 heartbeater.cc:503] Master 127.30.16.126:42645 was elected leader, sending a full tablet report...
I20260927 07:11:12.229455 15969 consensus_queue.cc:260] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.231695 15970 ts_tablet_manager.cc:1428] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
I20260927 07:11:12.233071 15969 ts_tablet_manager.cc:1428] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Time spent starting tablet: real 0.016s user 0.011s sys 0.005s
W20260927 07:11:12.250916 15682 tablet.cc:2369] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:12.307554 15816 tablet.cc:2369] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:12.340763 15950 tablet.cc:2369] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:12.360013 15976 raft_consensus.cc:482] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:12.360345 15976 raft_consensus.cc:504] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.362068 15976 leader_election.cc:288] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec956958d9754431af14d29ec76e0b52 (127.30.16.66:45697), dc44e3dff1db4d8e9065fdfd77eacb0d (127.30.16.67:36935)
I20260927 07:11:12.371668 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.362628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:34659 (local address 127.30.16.66:45697)
0927 07:11:12.362846 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.362853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.363640 (+ 787us) server_negotiation.cc:407] Connection header received
0927 07:11:12.363855 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.363860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.363940 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.364042 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:12.364945 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.366095 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.369226 (+ 3131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.370520 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.370735 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.370813 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.371231 (+ 418us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.371385 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:12.371819 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.362627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:38825 (local address 127.30.16.67:36935)
0927 07:11:12.362846 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.362853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.363640 (+ 787us) server_negotiation.cc:407] Connection header received
0927 07:11:12.363852 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.363857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.363940 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.364042 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:12.364984 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.366142 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.369240 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.370608 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.370751 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.370815 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.371424 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.371570 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:12.371814 15978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.362479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:45697 (local address 127.30.16.65:34659)
0927 07:11:12.363494 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.363516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.363708 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.364081 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.364086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.364124 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:12.364694 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.364714 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.366252 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.366258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.369036 (+ 2778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.369051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.370768 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.370774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.370964 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.370992 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.371169 (+ 177us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.371338 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":214,"spinlock_wait_cycles":26624,"thread_start_us":598,"threads_started":1}
I20260927 07:11:12.371959 15977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.362423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:36935 (local address 127.30.16.65:38825)
0927 07:11:12.363494 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:12.363516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:12.363682 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:12.364081 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:12.364086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:12.364145 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:12.364814 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.364828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.366257 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.366261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.369036 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:12.369051 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.371043 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.371050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:12.371253 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.371285 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:11:12.371375 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 07:11:12.371503 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":248,"spinlock_wait_cycles":48384,"thread_start_us":647,"threads_started":1}
I20260927 07:11:12.373059 15770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ca6528ce1c406b9a4823b7d0c70d7c" candidate_uuid: "60bc382cef4041ef9623bf0a18bb5d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec956958d9754431af14d29ec76e0b52" is_pre_election: true
I20260927 07:11:12.373064 15904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ca6528ce1c406b9a4823b7d0c70d7c" candidate_uuid: "60bc382cef4041ef9623bf0a18bb5d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" is_pre_election: true
I20260927 07:11:12.373595 15770 raft_consensus.cc:2454] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bc382cef4041ef9623bf0a18bb5d27 in term 0.
I20260927 07:11:12.373612 15904 raft_consensus.cc:2454] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60bc382cef4041ef9623bf0a18bb5d27 in term 0.
I20260927 07:11:12.374342 15571 leader_election.cc:302] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60bc382cef4041ef9623bf0a18bb5d27, dc44e3dff1db4d8e9065fdfd77eacb0d, ec956958d9754431af14d29ec76e0b52; no voters:
I20260927 07:11:12.374843 15976 raft_consensus.cc:2781] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:12.375001 15976 raft_consensus.cc:482] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:12.375108 15976 raft_consensus.cc:3037] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:12.377270 15976 raft_consensus.cc:504] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.377982 15976 leader_election.cc:288] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [CANDIDATE]: Term 1 election: Requested vote from peers ec956958d9754431af14d29ec76e0b52 (127.30.16.66:45697), dc44e3dff1db4d8e9065fdfd77eacb0d (127.30.16.67:36935)
I20260927 07:11:12.378489 15770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ca6528ce1c406b9a4823b7d0c70d7c" candidate_uuid: "60bc382cef4041ef9623bf0a18bb5d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec956958d9754431af14d29ec76e0b52"
I20260927 07:11:12.378593 15904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ca6528ce1c406b9a4823b7d0c70d7c" candidate_uuid: "60bc382cef4041ef9623bf0a18bb5d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d"
I20260927 07:11:12.378748 15770 raft_consensus.cc:3037] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:12.378880 15904 raft_consensus.cc:3037] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:12.381057 15770 raft_consensus.cc:2454] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bc382cef4041ef9623bf0a18bb5d27 in term 1.
I20260927 07:11:12.381230 15904 raft_consensus.cc:2454] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60bc382cef4041ef9623bf0a18bb5d27 in term 1.
I20260927 07:11:12.381518 15571 leader_election.cc:302] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60bc382cef4041ef9623bf0a18bb5d27, ec956958d9754431af14d29ec76e0b52; no voters:
I20260927 07:11:12.381852 15976 raft_consensus.cc:2781] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:12.382277 15976 raft_consensus.cc:686] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 1 LEADER]: Becoming Leader. State: Replica: 60bc382cef4041ef9623bf0a18bb5d27, State: Running, Role: LEADER
I20260927 07:11:12.382869 15976 consensus_queue.cc:237] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } }
I20260927 07:11:12.387491 15490 catalog_manager.cc:5495] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60bc382cef4041ef9623bf0a18bb5d27 (127.30.16.65). New cstate: current_term: 1 leader_uuid: "60bc382cef4041ef9623bf0a18bb5d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60bc382cef4041ef9623bf0a18bb5d27" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 43157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 } health_report { overall_health: UNKNOWN } } }
I20260927 07:11:12.430322 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.424414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55644 (local address 127.30.16.65:43157)
0927 07:11:12.424668 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.424674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.424742 (+ 68us) server_negotiation.cc:407] Connection header received
0927 07:11:12.424950 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.424954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.425033 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.425165 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:12.425585 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.426275 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.427954 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.428366 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.428512 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.429128 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:12.429688 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:12.429853 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.429954 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.430039 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260927 07:11:12.439210 15770 raft_consensus.cc:1261] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 1 FOLLOWER]: Refusing update from remote peer 60bc382cef4041ef9623bf0a18bb5d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:12.439206 15904 raft_consensus.cc:1261] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 1 FOLLOWER]: Refusing update from remote peer 60bc382cef4041ef9623bf0a18bb5d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:12.440199 15979 consensus_queue.cc:1035] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec956958d9754431af14d29ec76e0b52" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:12.440608 15976 consensus_queue.cc:1035] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc44e3dff1db4d8e9065fdfd77eacb0d" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:12.452831 15982 mvcc.cc:205] Tried to move back new op lower bound from 7333859624699490304 to 7333859624483844096. Current Snapshot: MvccSnapshot[applied={T|T < 7333859624699490304}]
I20260927 07:11:12.470073 15492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:12.470378 15492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:12.474303 15492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2473316ce9e3459097659bf8ef7130bd: Sending DeleteTablet for 3 replicas of tablet 34ca6528ce1c406b9a4823b7d0c70d7c
I20260927 07:11:12.475806 15616 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:12 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:55638
I20260927 07:11:12.475801 15750 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:12 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:43210
I20260927 07:11:12.475821 15884 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ca6528ce1c406b9a4823b7d0c70d7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:12 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:45706
I20260927 07:11:12.481096 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15553
I20260927 07:11:12.488655 15992 tablet_replica.cc:330] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: stopping tablet replica
I20260927 07:11:12.488978 15994 tablet_replica.cc:330] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: stopping tablet replica
I20260927 07:11:12.489746 15994 raft_consensus.cc:2229] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:12.490015 15992 raft_consensus.cc:2229] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:12.490808 15994 raft_consensus.cc:2258] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:12.491271 15992 raft_consensus.cc:2258] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:12.494246 15992 ts_tablet_manager.cc:1905] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:12.494594 15994 ts_tablet_manager.cc:1905] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:12.495694 15993 tablet_replica.cc:330] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: stopping tablet replica
I20260927 07:11:12.496277 15993 raft_consensus.cc:2229] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:12.496894 15993 raft_consensus.cc:2258] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 07:11:12 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:11:12.499248 15993 ts_tablet_manager.cc:1905] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:12.502506 15994 ts_tablet_manager.cc:1918] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:12.502800 15994 log.cc:1196] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/34ca6528ce1c406b9a4823b7d0c70d7c
I20260927 07:11:12.503499 15994 ts_tablet_manager.cc:1939] T 34ca6528ce1c406b9a4823b7d0c70d7c P 60bc382cef4041ef9623bf0a18bb5d27: Deleting consensus metadata
I20260927 07:11:12.505707 15477 catalog_manager.cc:4841] TS 60bc382cef4041ef9623bf0a18bb5d27 (127.30.16.65:43157): tablet 34ca6528ce1c406b9a4823b7d0c70d7c (table test-table [id=4afd87485f0d4d80bb53de97bc27b72e]) successfully deleted
I20260927 07:11:12.506495 15993 ts_tablet_manager.cc:1918] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:12.506793 15993 log.cc:1196] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/34ca6528ce1c406b9a4823b7d0c70d7c
I20260927 07:11:12.507428 15993 ts_tablet_manager.cc:1939] T 34ca6528ce1c406b9a4823b7d0c70d7c P ec956958d9754431af14d29ec76e0b52: Deleting consensus metadata
I20260927 07:11:12.508211 15992 ts_tablet_manager.cc:1918] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:12.508435 15992 log.cc:1196] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/34ca6528ce1c406b9a4823b7d0c70d7c
I20260927 07:11:12.509042 15992 ts_tablet_manager.cc:1939] T 34ca6528ce1c406b9a4823b7d0c70d7c P dc44e3dff1db4d8e9065fdfd77eacb0d: Deleting consensus metadata
I20260927 07:11:12.509603 15478 catalog_manager.cc:4841] TS ec956958d9754431af14d29ec76e0b52 (127.30.16.66:45697): tablet 34ca6528ce1c406b9a4823b7d0c70d7c (table test-table [id=4afd87485f0d4d80bb53de97bc27b72e]) successfully deleted
I20260927 07:11:12.510828 15479 catalog_manager.cc:4841] TS dc44e3dff1db4d8e9065fdfd77eacb0d (127.30.16.67:36935): tablet 34ca6528ce1c406b9a4823b7d0c70d7c (table test-table [id=4afd87485f0d4d80bb53de97bc27b72e]) successfully deleted
I20260927 07:11:12.513768 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.484215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.30.16.65:43157)
0927 07:11:12.484685 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.484692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.484712 (+ 20us) server_negotiation.cc:407] Connection header received
0927 07:11:12.484788 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.484794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.484908 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.485026 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:12.490097 (+ 5071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.491456 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.493543 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.494274 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.494379 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.506267 (+ 11888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:12.506288 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:12.506307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:12.506374 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:12.510604 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.511553 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.511559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.511562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.511702 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.512256 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.512259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.512261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.512616 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:12.512852 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.513276 (+ 424us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.513457 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352}
I20260927 07:11:12.514483 15676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:12.636210 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15553
I20260927 07:11:12.655836 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15687
Sep 27 07:11:12 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:11:12.671988 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.656701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.30.16.66:45697)
0927 07:11:12.656878 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.656883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.656902 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:11:12.657004 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.657008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.657068 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.657218 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:12.658438 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.659324 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.660636 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.661026 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.661126 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.665832 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:12.665851 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:12.665868 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:12.665914 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:12.669074 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.669835 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.669843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.669846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.669983 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.670375 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.670379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.670382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.670891 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:12.671132 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.671431 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.671601 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:12.672724 15810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:12.766964 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15687
I20260927 07:11:12.784395 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15821
Sep 27 07:11:12 dist-test-slave-8vqf krb5kdc[15447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:11:12.799193 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:12.785256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.30.16.67:36935)
0927 07:11:12.785433 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:12.785438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:12.785455 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:11:12.785517 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:12.785521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:12.785585 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:12.785669 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:12.787097 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.787828 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.789041 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:12.789418 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:12.789509 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:12.793929 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:12.793946 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:12.793961 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:12.794001 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:12.796752 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.797409 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.797416 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.797418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.797543 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:12.797881 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:12.797884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:12.797886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:12.798219 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:12.798441 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:12.798755 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 07:11:12.798930 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:11:12.800001 15944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:12.890363 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15821
I20260927 07:11:12.907428 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 15460
I20260927 07:11:12.908631 15522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:13.021459 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 15460
2026-09-27T07:11:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4609 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T07:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:13Z Disabled control of system clock
I20260927 07:11:13.100328 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33313
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:13.448760 16003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:13.449092 16003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:13.449177 16003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:13.456897 16003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:13.457016 16003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:13.457054 16003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:13.457108 16003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:13.457152 16003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:13.466814 16003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33313
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:13.468186 16003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:13.470002 16003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:13.482992 16011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:13.483069 16009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:13.483040 16008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:13.484020 16003 server_base.cc:1034] running on GCE node
I20260927 07:11:13.484927 16003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:13.486461 16003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:13.487751 16003 hybrid_clock.cc:648] HybridClock initialized: now 1790493073487700 us; error 44 us; skew 500 ppm
I20260927 07:11:13.488175 16003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:13.490828 16003 webserver.cc:460] Webserver started at http://127.30.16.126:34101/ using document root <none> and password file <none>
I20260927 07:11:13.491585 16003 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:13.491724 16003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:13.492074 16003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:13.495263 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "8dc293025ad04966afa220637fc599b1"
format_stamp: "Formatted at 2026-09-27 07:11:13 on dist-test-slave-8vqf"
server_key: "557ab55d4fea184a7695b817aa0da17b"
server_key_iv: "11a2744f4ea70fab62bf7a8a7b2b8916"
server_key_version: "encryptionkey@0"
I20260927 07:11:13.496105 16003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "8dc293025ad04966afa220637fc599b1"
format_stamp: "Formatted at 2026-09-27 07:11:13 on dist-test-slave-8vqf"
server_key: "557ab55d4fea184a7695b817aa0da17b"
server_key_iv: "11a2744f4ea70fab62bf7a8a7b2b8916"
server_key_version: "encryptionkey@0"
I20260927 07:11:13.501931 16003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:11:13.505730 16017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:13.507277 16003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:13.507483 16003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "8dc293025ad04966afa220637fc599b1"
format_stamp: "Formatted at 2026-09-27 07:11:13 on dist-test-slave-8vqf"
server_key: "557ab55d4fea184a7695b817aa0da17b"
server_key_iv: "11a2744f4ea70fab62bf7a8a7b2b8916"
server_key_version: "encryptionkey@0"
I20260927 07:11:13.507718 16003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:13.521790 16003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:13.522751 16003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:13.523082 16003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:13.542678 16003 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33313
I20260927 07:11:13.542697 16068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33313 every 8 connection(s)
I20260927 07:11:13.544135 16003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:13.545627 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16003
I20260927 07:11:13.545800 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:13.546396 30785 external_mini_cluster.cc:1428] Setting key 7f509f7765c032605cbf923d80278b51
I20260927 07:11:13.549217 16069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:13.561071 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:13.548475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.30.16.126:33313)
0927 07:11:13.549941 (+ 1466us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:13.549954 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:13.550001 (+ 47us) server_negotiation.cc:407] Connection header received
0927 07:11:13.551685 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:13.551714 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:13.552217 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:13.552637 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:13.553228 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:13.554671 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:13.556190 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:13.557013 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:13.557231 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:13.558520 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:13.558532 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:13.558592 (+ 60us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:13.559227 (+ 635us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:13.559312 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:13.559334 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:13.559469 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:13.559714 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 07:11:13.560057 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":521,"threads_started":1}
I20260927 07:11:13.561179 16069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: Bootstrap starting.
I20260927 07:11:13.564993 16069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:13.566087 16069 log.cc:824] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:13.569186 16069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: No bootstrap required, opened a new log
I20260927 07:11:13.576997 16069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } }
I20260927 07:11:13.577438 16069 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:13.577549 16069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc293025ad04966afa220637fc599b1, State: Initialized, Role: FOLLOWER
I20260927 07:11:13.578145 16069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } }
I20260927 07:11:13.578392 16069 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:13.578594 16069 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:13.578814 16069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:13.581571 16069 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } }
I20260927 07:11:13.582108 16069 leader_election.cc:302] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8dc293025ad04966afa220637fc599b1; no voters:
I20260927 07:11:13.582788 16069 leader_election.cc:288] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:13.582957 16074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:13.583701 16074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [term 1 LEADER]: Becoming Leader. State: Replica: 8dc293025ad04966afa220637fc599b1, State: Running, Role: LEADER
I20260927 07:11:13.584278 16074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } }
I20260927 07:11:13.584977 16069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:13.587457 16075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8dc293025ad04966afa220637fc599b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } } }
I20260927 07:11:13.588042 16075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:13.588016 16076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dc293025ad04966afa220637fc599b1. Latest consensus state: current_term: 1 leader_uuid: "8dc293025ad04966afa220637fc599b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8dc293025ad04966afa220637fc599b1" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33313 } } }
I20260927 07:11:13.588318 16076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:13.590181 16082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:13.595475 16082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:13.606611 16082 catalog_manager.cc:1347] Generated new cluster ID: 615704047d4e417391bbffb40ad7269a
I20260927 07:11:13.606752 16082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:13.636173 16082 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:13.637531 16082 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:13.646976 16082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: Generated new TSK 0
I20260927 07:11:13.647846 16082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:13.659468 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--builtin_ntp_servers=127.30.16.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:14.008773 16093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:14.009161 16093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:14.009326 16093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:14.017261 16093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:14.017441 16093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:14.017619 16093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:14.027251 16093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:14.028618 16093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:14.030459 16093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:14.044231 16099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.044262 16101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.044302 16098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:14.045483 16093 server_base.cc:1034] running on GCE node
I20260927 07:11:14.046141 16093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:14.047297 16093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:14.048507 16093 hybrid_clock.cc:648] HybridClock initialized: now 1790493074048455 us; error 46 us; skew 500 ppm
I20260927 07:11:14.048861 16093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:14.051268 16093 webserver.cc:460] Webserver started at http://127.30.16.65:35921/ using document root <none> and password file <none>
I20260927 07:11:14.051994 16093 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:14.052106 16093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:14.052441 16093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:14.055454 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "d005b92d63144e74894dc49c49148d66"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "2399705d9f16c6b80a9cf64b79ad10c2"
server_key_iv: "4824d5a974a0566bd71a020a5d36de10"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.056272 16093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "d005b92d63144e74894dc49c49148d66"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "2399705d9f16c6b80a9cf64b79ad10c2"
server_key_iv: "4824d5a974a0566bd71a020a5d36de10"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.061779 16093 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:11:14.065569 16107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.067044 16093 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 07:11:14.067221 16093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "d005b92d63144e74894dc49c49148d66"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "2399705d9f16c6b80a9cf64b79ad10c2"
server_key_iv: "4824d5a974a0566bd71a020a5d36de10"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.067437 16093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:14.102409 16093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:14.103356 16093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:14.103668 16093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:14.104672 16093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:14.106172 16093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:14.106299 16093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.106438 16093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:14.106530 16093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.140089 16093 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:35561
I20260927 07:11:14.140180 16219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:35561 every 8 connection(s)
I20260927 07:11:14.141458 16093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:14.143493 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16093
I20260927 07:11:14.143676 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:14.144131 30785 external_mini_cluster.cc:1428] Setting key 09b35a77b53cec9220b6dc6153873ae8
I20260927 07:11:14.147259 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--builtin_ntp_servers=127.30.16.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:14.155908 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:14.144323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:58653 (local address 127.30.16.126:33313)
0927 07:11:14.144816 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:14.144822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:14.146384 (+ 1562us) server_negotiation.cc:407] Connection header received
0927 07:11:14.148459 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:14.148468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:14.148560 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:14.148682 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:14.149869 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.151029 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.152568 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.153301 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.153460 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:14.154884 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:14.154893 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:14.154900 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:14.154928 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:14.154980 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:14.154999 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:14.155098 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:14.155325 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 07:11:14.155446 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260927 07:11:14.156819 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:14.145048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33313 (local address 127.30.16.65:58653)
0927 07:11:14.146211 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:14.146262 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:14.148180 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:14.148877 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:14.148885 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:14.149015 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:14.149695 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:14.149713 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.151173 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.151177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:14.152419 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:14.152430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.153520 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.153524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:14.153719 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:14.154486 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:14.154489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:14.154529 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:14.154669 (+ 140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:14.155107 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:14.155115 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:14.155121 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:11:14.155258 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 07:11:14.155572 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":357,"threads_started":1}
I20260927 07:11:14.158533 16220 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33313
I20260927 07:11:14.158928 16220 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:14.159695 16220 heartbeater.cc:511] Master 127.30.16.126:33313 requested a full tablet report, sending...
I20260927 07:11:14.161900 16034 ts_manager.cc:194] Registered new tserver with Master: d005b92d63144e74894dc49c49148d66 (127.30.16.65:35561)
I20260927 07:11:14.163720 16034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.65:58653
W20260927 07:11:14.485618 16224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:14.485920 16224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:14.486008 16224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:14.493829 16224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:14.493945 16224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:14.494053 16224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:14.503352 16224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:14.504671 16224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:14.506477 16224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:14.520401 16232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.520443 16230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.520464 16229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:14.521271 16224 server_base.cc:1034] running on GCE node
I20260927 07:11:14.521934 16224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:14.523186 16224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:14.524417 16224 hybrid_clock.cc:648] HybridClock initialized: now 1790493074524346 us; error 63 us; skew 500 ppm
I20260927 07:11:14.524715 16224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:14.527118 16224 webserver.cc:460] Webserver started at http://127.30.16.66:40381/ using document root <none> and password file <none>
I20260927 07:11:14.527951 16224 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:14.528040 16224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:14.528322 16224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:14.531365 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "fea74826847843e9b5c9fb044120ee38"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "8865224d9793c6d3a42a64fab5661f0f"
server_key_iv: "b9b0b76fd44781858ab796f32276c1c0"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.532117 16224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "fea74826847843e9b5c9fb044120ee38"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "8865224d9793c6d3a42a64fab5661f0f"
server_key_iv: "b9b0b76fd44781858ab796f32276c1c0"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.539963 16224 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20260927 07:11:14.544329 16238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.545765 16224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:14.545948 16224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "fea74826847843e9b5c9fb044120ee38"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "8865224d9793c6d3a42a64fab5661f0f"
server_key_iv: "b9b0b76fd44781858ab796f32276c1c0"
server_key_version: "encryptionkey@0"
I20260927 07:11:14.546183 16224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:14.564960 16224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:14.565873 16224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:14.566197 16224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:14.567292 16224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:14.568775 16224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:14.568902 16224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.569037 16224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:14.569124 16224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:14.601460 16224 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:43773
I20260927 07:11:14.601507 16350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:43773 every 8 connection(s)
I20260927 07:11:14.602945 16224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:14.604516 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16224
I20260927 07:11:14.604702 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:14.609642 30785 external_mini_cluster.cc:1428] Setting key a24f0867bdb9ecf98e004ed09f4c3525
I20260927 07:11:14.612607 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--builtin_ntp_servers=127.30.16.84:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:14.616617 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:14.605913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:40471 (local address 127.30.16.126:33313)
0927 07:11:14.606097 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:14.606101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:14.607958 (+ 1857us) server_negotiation.cc:407] Connection header received
0927 07:11:14.609769 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:14.609774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:14.609876 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:14.609989 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:14.611217 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.612005 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.613753 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.614165 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.614322 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:14.615797 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:14.615802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:14.615806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:14.615824 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:14.615854 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:14.615867 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:14.615963 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:14.616194 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 07:11:14.616304 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:11:14.617662 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:14.606766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33313 (local address 127.30.16.66:40471)
0927 07:11:14.607781 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:14.607818 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:14.609484 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:14.610226 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:14.610237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:14.610330 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:14.611051 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:14.611070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.612183 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.612188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:14.613601 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:14.613613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:14.614410 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:14.614417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:14.614716 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:14.615462 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:14.615464 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:14.615506 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:14.615593 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:14.615961 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:14.615974 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:14.615984 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:11:14.616168 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 07:11:14.616581 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":319,"threads_started":1}
I20260927 07:11:14.619208 16351 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33313
I20260927 07:11:14.619547 16351 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:14.620335 16351 heartbeater.cc:511] Master 127.30.16.126:33313 requested a full tablet report, sending...
I20260927 07:11:14.621953 16034 ts_manager.cc:194] Registered new tserver with Master: fea74826847843e9b5c9fb044120ee38 (127.30.16.66:43773)
I20260927 07:11:14.623260 16034 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.66:40471
W20260927 07:11:14.955355 16354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:14.955714 16354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:14.955883 16354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:14.963781 16354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:14.963925 16354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:14.964107 16354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:14.973874 16354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:14.975333 16354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:14.977131 16354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:14.990711 16359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.990739 16360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:14.990779 16362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:14.991811 16354 server_base.cc:1034] running on GCE node
I20260927 07:11:14.992506 16354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:14.993548 16354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:14.994777 16354 hybrid_clock.cc:648] HybridClock initialized: now 1790493074994720 us; error 54 us; skew 500 ppm
I20260927 07:11:14.995129 16354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:14.997485 16354 webserver.cc:460] Webserver started at http://127.30.16.67:46619/ using document root <none> and password file <none>
I20260927 07:11:14.998207 16354 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:14.998342 16354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:14.998737 16354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:15.001614 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "19981e098a2642adbae4750f24a90354"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "a5a30f3628c8fff335783d71960947c4"
server_key_iv: "47546809d5fd903241fcdf4e00ad9f7a"
server_key_version: "encryptionkey@0"
I20260927 07:11:15.002461 16354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "19981e098a2642adbae4750f24a90354"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "a5a30f3628c8fff335783d71960947c4"
server_key_iv: "47546809d5fd903241fcdf4e00ad9f7a"
server_key_version: "encryptionkey@0"
I20260927 07:11:15.008086 16354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 07:11:15.011970 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:15.013613 16354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:15.013811 16354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "19981e098a2642adbae4750f24a90354"
format_stamp: "Formatted at 2026-09-27 07:11:14 on dist-test-slave-8vqf"
server_key: "a5a30f3628c8fff335783d71960947c4"
server_key_iv: "47546809d5fd903241fcdf4e00ad9f7a"
server_key_version: "encryptionkey@0"
I20260927 07:11:15.014045 16354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:15.036736 16354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:15.037632 16354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:15.037967 16354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:15.039171 16354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:15.040887 16354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:15.041011 16354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:15.041142 16354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:15.041230 16354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:15.076140 16354 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:45391
I20260927 07:11:15.076231 16480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:45391 every 8 connection(s)
I20260927 07:11:15.077590 16354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:15.078182 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16354
I20260927 07:11:15.078383 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:15.078931 30785 external_mini_cluster.cc:1428] Setting key 8f89251c02e2d5d91f52175bbc236dee
I20260927 07:11:15.091306 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.080758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:44379 (local address 127.30.16.126:33313)
0927 07:11:15.080993 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.081002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.083139 (+ 2137us) server_negotiation.cc:407] Connection header received
0927 07:11:15.085035 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.085040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.085110 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.085193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:15.086338 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.087128 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.088645 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.089060 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.089185 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.090524 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:15.090528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:15.090532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:15.090595 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:15.090639 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:15.090653 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:15.090742 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.090964 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.091047 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:11:15.092308 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.081523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33313 (local address 127.30.16.67:44379)
0927 07:11:15.082977 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.083036 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.084748 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.085438 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.085451 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.085562 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:15.086188 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.086206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.087267 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.087271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.088519 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.088531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.089286 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.089290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.089507 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.090168 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:15.090170 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:15.090227 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:15.090348 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:15.090760 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:15.090773 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:15.090779 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.090936 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.091241 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":386,"threads_started":1}
I20260927 07:11:15.093716 16481 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33313
I20260927 07:11:15.094043 16481 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:15.094926 16481 heartbeater.cc:511] Master 127.30.16.126:33313 requested a full tablet report, sending...
I20260927 07:11:15.096338 16033 ts_manager.cc:194] Registered new tserver with Master: 19981e098a2642adbae4750f24a90354 (127.30.16.67:45391)
I20260927 07:11:15.097215 16033 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.67:44379
I20260927 07:11:15.106837 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:15.116875 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.110869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.30.16.126:33313)
0927 07:11:15.111083 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.111092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.111480 (+ 388us) server_negotiation.cc:407] Connection header received
0927 07:11:15.111702 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.111711 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.111789 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.111900 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:15.112422 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.113139 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.114517 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.114976 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.115120 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.116115 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:15.116126 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:15.116130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:15.116148 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:15.116181 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:15.116194 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:15.116289 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.116451 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.116550 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:11:15.121243 16033 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:15.128374 16033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:15.161571 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.140836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:43773 (local address 127.0.0.1:33194)
0927 07:11:15.148924 (+ 8088us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.148964 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.149151 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.150822 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.150833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.150877 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.151404 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.151423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.154020 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.154029 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.157382 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.157409 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.160590 (+ 3181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.160595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.160832 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.160900 (+ 68us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.161025 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.161225 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7804,"threads_started":1}
I20260927 07:11:15.162876 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.141050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33194 (local address 127.30.16.66:43773)
0927 07:11:15.149754 (+ 8704us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.149763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.149793 (+ 30us) server_negotiation.cc:407] Connection header received
0927 07:11:15.149889 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.149898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.150177 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.150435 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.151748 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.153813 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.157754 (+ 3941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.159879 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.160057 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.160436 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.162267 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.162470 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8515,"thread_start_us":13227,"threads_started":1}
I20260927 07:11:15.166945 16220 heartbeater.cc:503] Master 127.30.16.126:33313 was elected leader, sending a full tablet report...
I20260927 07:11:15.167478 16286 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4634213e054a59abb86bc66f87fc8a (DEFAULT_TABLE table=test-table [id=33acd66f256a4a198bc83793e710ffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:15.169203 16286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4634213e054a59abb86bc66f87fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:15.169353 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.149048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:45391 (local address 127.0.0.1:50712)
0927 07:11:15.161900 (+ 12852us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.161921 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.162265 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.162991 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.162996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.163023 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.163241 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.163254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.164961 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.164966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.166649 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.166673 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.168673 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.168679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.168860 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.168896 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.168959 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.169054 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12747,"spinlock_wait_cycles":430336,"thread_start_us":11360,"threads_started":1}
I20260927 07:11:15.170246 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.149467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.30.16.67:45391)
0927 07:11:15.161696 (+ 12229us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.161709 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.162116 (+ 407us) server_negotiation.cc:407] Connection header received
0927 07:11:15.162453 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.162460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.162731 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.162974 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.163420 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.164777 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.166813 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.168142 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.168271 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.169647 (+ 1376us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.169711 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.169853 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11953,"thread_start_us":12243,"threads_started":1}
I20260927 07:11:15.171399 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.142700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.30.16.65:35561)
0927 07:11:15.148314 (+ 5614us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.148323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.161048 (+ 12725us) server_negotiation.cc:407] Connection header received
0927 07:11:15.162348 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.162358 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.162724 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.162880 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.163716 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.165135 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.168410 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.169877 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.169979 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.170238 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.170869 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.171034 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5443,"thread_start_us":5322,"threads_started":1}
I20260927 07:11:15.172120 16497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.142091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54086)
0927 07:11:15.160859 (+ 18768us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.160888 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.162093 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.163200 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.163205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.163226 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.163570 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.163578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.165328 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.165332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.168188 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.168206 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.170415 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.170421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.170654 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.170687 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.171615 (+ 928us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.171770 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18529,"mutex_wait_us":3,"spinlock_wait_cycles":4352,"thread_start_us":7185,"threads_started":1}
I20260927 07:11:15.174598 16416 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4634213e054a59abb86bc66f87fc8a (DEFAULT_TABLE table=test-table [id=33acd66f256a4a198bc83793e710ffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:15.175604 16155 tablet_service.cc:1467] Processing CreateTablet for tablet 4c4634213e054a59abb86bc66f87fc8a (DEFAULT_TABLE table=test-table [id=33acd66f256a4a198bc83793e710ffa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:15.176254 16416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4634213e054a59abb86bc66f87fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:15.177044 16155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c4634213e054a59abb86bc66f87fc8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:15.185909 16499 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Bootstrap starting.
I20260927 07:11:15.190130 16499 tablet_bootstrap.cc:654] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:15.190220 16500 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Bootstrap starting.
I20260927 07:11:15.191789 16501 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Bootstrap starting.
I20260927 07:11:15.191828 16499 log.cc:824] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:15.193975 16500 tablet_bootstrap.cc:654] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:15.195298 16500 log.cc:824] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:15.195441 16501 tablet_bootstrap.cc:654] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:15.196910 16501 log.cc:824] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:15.201385 16500 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: No bootstrap required, opened a new log
I20260927 07:11:15.201399 16499 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: No bootstrap required, opened a new log
I20260927 07:11:15.201391 16501 tablet_bootstrap.cc:492] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: No bootstrap required, opened a new log
I20260927 07:11:15.201714 16500 ts_tablet_manager.cc:1397] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260927 07:11:15.201725 16499 ts_tablet_manager.cc:1397] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260927 07:11:15.201726 16501 ts_tablet_manager.cc:1397] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 07:11:15.210390 16500 raft_consensus.cc:348] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.210912 16500 raft_consensus.cc:374] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:15.211074 16500 raft_consensus.cc:729] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19981e098a2642adbae4750f24a90354, State: Initialized, Role: FOLLOWER
I20260927 07:11:15.211854 16500 consensus_queue.cc:260] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.212719 16499 raft_consensus.cc:348] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.212726 16501 raft_consensus.cc:348] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.213120 16481 heartbeater.cc:503] Master 127.30.16.126:33313 was elected leader, sending a full tablet report...
I20260927 07:11:15.213158 16501 raft_consensus.cc:374] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:15.213200 16499 raft_consensus.cc:374] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:15.213264 16501 raft_consensus.cc:729] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d005b92d63144e74894dc49c49148d66, State: Initialized, Role: FOLLOWER
I20260927 07:11:15.213284 16499 raft_consensus.cc:729] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea74826847843e9b5c9fb044120ee38, State: Initialized, Role: FOLLOWER
I20260927 07:11:15.213912 16500 ts_tablet_manager.cc:1428] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260927 07:11:15.213939 16499 consensus_queue.cc:260] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.213938 16501 consensus_queue.cc:260] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.215917 16501 ts_tablet_manager.cc:1428] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 07:11:15.216161 16499 ts_tablet_manager.cc:1428] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Time spent starting tablet: real 0.014s user 0.013s sys 0.003s
I20260927 07:11:15.216426 16351 heartbeater.cc:503] Master 127.30.16.126:33313 was elected leader, sending a full tablet report...
W20260927 07:11:15.333669 16482 tablet.cc:2369] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:15.354976 16352 tablet.cc:2369] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:15.393967 16221 tablet.cc:2369] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:15.474839 16505 raft_consensus.cc:482] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:15.475176 16505 raft_consensus.cc:504] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.476985 16505 leader_election.cc:288] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fea74826847843e9b5c9fb044120ee38 (127.30.16.66:43773), d005b92d63144e74894dc49c49148d66 (127.30.16.65:35561)
I20260927 07:11:15.481537 16506 raft_consensus.cc:482] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:15.481842 16506 raft_consensus.cc:504] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.484273 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.477472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:44313 (local address 127.30.16.65:35561)
0927 07:11:15.477668 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.477677 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.477704 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:11:15.477906 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.477912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.478032 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.478140 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.478692 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.479627 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.482049 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.483282 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.483442 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.483514 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.483883 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.484015 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:11:15.484287 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.477401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.30.16.67:44313)
0927 07:11:15.477574 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.477595 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.477728 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.478274 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.478280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.478306 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.478518 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.478530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.479747 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.479759 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.481851 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.481885 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.483452 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.483458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.483641 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.483731 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.483856 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.484011 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":24}
I20260927 07:11:15.484529 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.477521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:59703 (local address 127.30.16.66:43773)
0927 07:11:15.477716 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.477721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.478588 (+ 867us) server_negotiation.cc:407] Connection header received
0927 07:11:15.478703 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.478709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.478799 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.478899 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.479388 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.480183 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.482519 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.483487 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.483637 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.483711 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.484088 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.484234 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 07:11:15.485638 16506 leader_election.cc:288] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d005b92d63144e74894dc49c49148d66 (127.30.16.65:35561), 19981e098a2642adbae4750f24a90354 (127.30.16.67:45391)
I20260927 07:11:15.485812 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "19981e098a2642adbae4750f24a90354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d005b92d63144e74894dc49c49148d66" is_pre_election: true
I20260927 07:11:15.486387 16508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.477414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:43773 (local address 127.30.16.67:59703)
0927 07:11:15.478424 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.478446 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.478594 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.478916 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.478918 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.478937 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.479224 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.479231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.480329 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.480333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.482341 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.482352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.483647 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.483654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.483898 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.483932 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.484755 (+ 823us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.484967 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":228,"thread_start_us":408,"threads_started":1}
I20260927 07:11:15.486529 16175 raft_consensus.cc:2454] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19981e098a2642adbae4750f24a90354 in term 0.
I20260927 07:11:15.487440 16369 leader_election.cc:302] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19981e098a2642adbae4750f24a90354, d005b92d63144e74894dc49c49148d66; no voters:
I20260927 07:11:15.487398 16306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "19981e098a2642adbae4750f24a90354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea74826847843e9b5c9fb044120ee38" is_pre_election: true
I20260927 07:11:15.487843 16306 raft_consensus.cc:2454] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19981e098a2642adbae4750f24a90354 in term 0.
I20260927 07:11:15.487948 16505 raft_consensus.cc:2781] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:15.488080 16505 raft_consensus.cc:482] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:15.488168 16505 raft_consensus.cc:3037] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:15.491508 16505 raft_consensus.cc:504] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.492719 16306 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "19981e098a2642adbae4750f24a90354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea74826847843e9b5c9fb044120ee38"
I20260927 07:11:15.492775 16175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "19981e098a2642adbae4750f24a90354" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d005b92d63144e74894dc49c49148d66"
I20260927 07:11:15.492949 16306 raft_consensus.cc:3037] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:15.493024 16175 raft_consensus.cc:3037] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:15.493537 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.484718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:42659 (local address 127.30.16.65:35561)
0927 07:11:15.485272 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.485280 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.485303 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:11:15.485363 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.485369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.485450 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.485533 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.486302 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.488138 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.490964 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.492224 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.492387 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.492460 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.492976 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.493175 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":1271,"threads_started":1}
I20260927 07:11:15.494323 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.484262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.30.16.66:42659)
0927 07:11:15.485059 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.485078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.485241 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.485681 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.485685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.485711 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.486127 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.486141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.488351 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.488356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.490763 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.490778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.492583 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.492590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.492770 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.492814 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.493783 (+ 969us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.493992 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":623,"threads_started":1}
I20260927 07:11:15.494367 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.486871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:34483 (local address 127.30.16.67:45391)
0927 07:11:15.487041 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.487046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.487566 (+ 520us) server_negotiation.cc:407] Connection header received
0927 07:11:15.487841 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.487846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.487926 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.487999 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:15.488914 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.489775 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.491996 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.493211 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.493392 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.493470 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.493980 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.494120 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:15.495028 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.486801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:45391 (local address 127.30.16.66:34483)
0927 07:11:15.487453 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.487476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:15.487669 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:15.488180 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:15.488184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:15.488212 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:15.488714 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.488729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.489916 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.489922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.491796 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:15.491812 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.493600 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.493607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:15.493791 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.493823 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 07:11:15.494516 (+ 693us) client_negotiation.cc:240] Negotiation successful
0927 07:11:15.494705 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":378,"threads_started":1}
I20260927 07:11:15.495059 16174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "fea74826847843e9b5c9fb044120ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d005b92d63144e74894dc49c49148d66" is_pre_election: true
I20260927 07:11:15.495976 16436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c4634213e054a59abb86bc66f87fc8a" candidate_uuid: "fea74826847843e9b5c9fb044120ee38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19981e098a2642adbae4750f24a90354" is_pre_election: true
I20260927 07:11:15.496497 16436 raft_consensus.cc:2379] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fea74826847843e9b5c9fb044120ee38 in current term 1: Already voted for candidate 19981e098a2642adbae4750f24a90354 in this term.
I20260927 07:11:15.496482 16306 raft_consensus.cc:2454] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19981e098a2642adbae4750f24a90354 in term 1.
I20260927 07:11:15.496570 16175 raft_consensus.cc:2454] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19981e098a2642adbae4750f24a90354 in term 1.
I20260927 07:11:15.496975 16370 leader_election.cc:302] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19981e098a2642adbae4750f24a90354, fea74826847843e9b5c9fb044120ee38; no voters:
I20260927 07:11:15.497059 16505 leader_election.cc:288] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [CANDIDATE]: Term 1 election: Requested vote from peers fea74826847843e9b5c9fb044120ee38 (127.30.16.66:43773), d005b92d63144e74894dc49c49148d66 (127.30.16.65:35561)
I20260927 07:11:15.497255 16505 raft_consensus.cc:2781] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:15.497503 16505 raft_consensus.cc:686] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 LEADER]: Becoming Leader. State: Replica: 19981e098a2642adbae4750f24a90354, State: Running, Role: LEADER
I20260927 07:11:15.497535 16242 leader_election.cc:302] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fea74826847843e9b5c9fb044120ee38; no voters: 19981e098a2642adbae4750f24a90354, d005b92d63144e74894dc49c49148d66
I20260927 07:11:15.497931 16506 raft_consensus.cc:2726] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 07:11:15.498087 16505 consensus_queue.cc:237] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } }
I20260927 07:11:15.502784 16034 catalog_manager.cc:5495] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19981e098a2642adbae4750f24a90354 (127.30.16.67). New cstate: current_term: 1 leader_uuid: "19981e098a2642adbae4750f24a90354" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19981e098a2642adbae4750f24a90354" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45391 } health_report { overall_health: HEALTHY } } }
I20260927 07:11:15.533663 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.528144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.30.16.67:45391)
0927 07:11:15.528348 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.528352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.528370 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:11:15.528532 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.528536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.528617 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.528755 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:15.529111 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.529932 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.531546 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.531942 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.532037 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.532541 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:15.533063 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:15.533181 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.533267 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.533348 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:11:15.541996 16175 raft_consensus.cc:1261] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 1 FOLLOWER]: Refusing update from remote peer 19981e098a2642adbae4750f24a90354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:15.542049 16306 raft_consensus.cc:1261] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 1 FOLLOWER]: Refusing update from remote peer 19981e098a2642adbae4750f24a90354: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:15.543035 16512 consensus_queue.cc:1035] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea74826847843e9b5c9fb044120ee38" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 43773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:15.543422 16505 consensus_queue.cc:1035] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d005b92d63144e74894dc49c49148d66" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:15.582499 16034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:15.582933 16034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:15.586872 16034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8dc293025ad04966afa220637fc599b1: Sending DeleteTablet for 3 replicas of tablet 4c4634213e054a59abb86bc66f87fc8a
I20260927 07:11:15.588054 16416 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4634213e054a59abb86bc66f87fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:15 UTC) from {username='slave'} at 127.0.0.1:50712
I20260927 07:11:15.588049 16155 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4634213e054a59abb86bc66f87fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:15 UTC) from {username='slave'} at 127.0.0.1:54086
I20260927 07:11:15.588091 16286 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c4634213e054a59abb86bc66f87fc8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:15 UTC) from {username='slave'} at 127.0.0.1:33194
I20260927 07:11:15.593631 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16093
I20260927 07:11:15.603564 16527 tablet_replica.cc:330] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: stopping tablet replica
I20260927 07:11:15.604964 16527 raft_consensus.cc:2229] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:15.606335 16527 raft_consensus.cc:2258] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:15.610101 16527 ts_tablet_manager.cc:1905] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:15.619302 16526 tablet_replica.cc:330] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: stopping tablet replica
I20260927 07:11:15.620738 16526 raft_consensus.cc:2229] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:15.623164 16526 raft_consensus.cc:2258] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:15.626526 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.595565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.30.16.65:35561)
0927 07:11:15.595823 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.595834 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.613909 (+ 18075us) server_negotiation.cc:407] Connection header received
0927 07:11:15.614439 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.614446 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.614615 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.614705 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:15.615401 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.616612 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.621033 (+ 4421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.621669 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.621774 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.624535 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:15.624553 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:15.624579 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:15.625635 (+ 1056us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:15.625701 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:15.625728 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:15.625830 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.626068 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.626192 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 07:11:15.628079 16215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:15.630281 16526 ts_tablet_manager.cc:1905] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:15.631672 16527 ts_tablet_manager.cc:1918] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:15.631860 16527 log.cc:1196] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/4c4634213e054a59abb86bc66f87fc8a
I20260927 07:11:15.632462 16527 ts_tablet_manager.cc:1939] T 4c4634213e054a59abb86bc66f87fc8a P fea74826847843e9b5c9fb044120ee38: Deleting consensus metadata
I20260927 07:11:15.634646 16019 catalog_manager.cc:4841] TS fea74826847843e9b5c9fb044120ee38 (127.30.16.66:43773): tablet 4c4634213e054a59abb86bc66f87fc8a (table test-table [id=33acd66f256a4a198bc83793e710ffa9]) successfully deleted
I20260927 07:11:15.641742 16526 ts_tablet_manager.cc:1918] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:15.642757 16526 log.cc:1196] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/4c4634213e054a59abb86bc66f87fc8a
I20260927 07:11:15.643425 16526 ts_tablet_manager.cc:1939] T 4c4634213e054a59abb86bc66f87fc8a P 19981e098a2642adbae4750f24a90354: Deleting consensus metadata
I20260927 07:11:15.645532 16021 catalog_manager.cc:4841] TS 19981e098a2642adbae4750f24a90354 (127.30.16.67:45391): tablet 4c4634213e054a59abb86bc66f87fc8a (table test-table [id=33acd66f256a4a198bc83793e710ffa9]) successfully deleted
I20260927 07:11:15.739354 16525 tablet_replica.cc:330] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66: stopping tablet replica
I20260927 07:11:15.739787 16525 raft_consensus.cc:2229] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:15.740270 16525 raft_consensus.cc:2258] T 4c4634213e054a59abb86bc66f87fc8a P d005b92d63144e74894dc49c49148d66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:15.741137 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16093
W20260927 07:11:15.759222 16018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.30.16.65:35561 (error 108)
I20260927 07:11:15.760965 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16224
I20260927 07:11:15.761457 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.760616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54108)
0927 07:11:15.761014 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.761222 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"threads_started":1}
W20260927 07:11:15.761837 16018 catalog_manager.cc:4584] TS d005b92d63144e74894dc49c49148d66 (127.30.16.65:35561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c4634213e054a59abb86bc66f87fc8a: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
I20260927 07:11:15.769040 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.761864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.30.16.66:43773)
0927 07:11:15.762266 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.762272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.762295 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:11:15.762365 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.762370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.762493 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.762653 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:15.763045 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.764150 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.765445 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.766118 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.766248 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.767409 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:15.767422 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:15.767442 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:15.768252 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:15.768321 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:15.768348 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:15.768463 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.768550 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.768668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273}
I20260927 07:11:15.769804 16346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:15.815836 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.815281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54120)
0927 07:11:15.815526 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.815641 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260927 07:11:15.817131 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.816657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54130)
0927 07:11:15.816846 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.816942 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 07:11:15.862602 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16224
I20260927 07:11:15.877909 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.877275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54136)
0927 07:11:15.877573 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.877698 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 07:11:15.879276 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.878793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54146)
0927 07:11:15.878983 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.879084 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 07:11:15.881309 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16354
I20260927 07:11:15.889720 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.882187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.30.16.67:45391)
0927 07:11:15.882386 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:15.882392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:15.882420 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:11:15.882616 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:15.882622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:15.882753 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:15.882894 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:15.883257 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.884427 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.885795 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:15.886720 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:15.886872 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:15.887928 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:15.887940 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:15.887960 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:15.888855 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:15.888921 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:15.888947 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:15.889059 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:15.889159 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 07:11:15.889283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 07:11:15.890517 16476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:15.960381 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.959870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54160)
0927 07:11:15.960102 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.960215 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 07:11:15.961791 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:15.961365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54164)
0927 07:11:15.961565 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:15.961641 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 07:11:16.023324 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16354
I20260927 07:11:16.043135 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16003
I20260927 07:11:16.044183 16064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:16.154976 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:16.154356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:35561 (local address 127.0.0.1:54176)
0927 07:11:16.154544 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:16.154696 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.65:35561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 07:11:16.154975 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16003
2026-09-27T07:11:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3135 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:16 dist-test-slave-8vqf krb5kdc[16537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:16 dist-test-slave-8vqf krb5kdc[16537](info): set up 2 sockets
Sep 27 07:11:16 dist-test-slave-8vqf krb5kdc[16537](info): commencing operation
krb5kdc: starting...
W20260927 07:11:17.299026 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:17 dist-test-slave-8vqf krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:11:17.562712 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43547
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:43547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:17.905916 16550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:17.906291 16550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:17.906447 16550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:17.913882 16550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:17.914052 16550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:17.914160 16550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:17.914259 16550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:17.914341 16550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:17.924564 16550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:43547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:43547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:17.926069 16550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:17.927495 16550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:17.939505 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:17.939590 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:17.939554 16558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:17.940634 16550 server_base.cc:1034] running on GCE node
I20260927 07:11:17.941480 16550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:17.943101 16550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:17.944339 16550 hybrid_clock.cc:648] HybridClock initialized: now 1790493077944284 us; error 46 us; skew 500 ppm
I20260927 07:11:17.944696 16550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:17 dist-test-slave-8vqf krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:17.952762 16550 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:11:17.954461 16550 webserver.cc:460] Webserver started at http://127.30.16.126:41957/ using document root <none> and password file <none>
I20260927 07:11:17.955255 16550 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:17.955406 16550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:17.955717 16550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:17.959008 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "b6820f423db64b6285a60f6e7f472344"
format_stamp: "Formatted at 2026-09-27 07:11:17 on dist-test-slave-8vqf"
server_key: "c584b1ed36be9521a2e00f368f1303f9"
server_key_iv: "a89904fb7ed7b977d36c631055610cf0"
server_key_version: "encryptionkey@0"
I20260927 07:11:17.959832 16550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "b6820f423db64b6285a60f6e7f472344"
format_stamp: "Formatted at 2026-09-27 07:11:17 on dist-test-slave-8vqf"
server_key: "c584b1ed36be9521a2e00f368f1303f9"
server_key_iv: "a89904fb7ed7b977d36c631055610cf0"
server_key_version: "encryptionkey@0"
I20260927 07:11:17.965358 16550 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:11:17.969151 16565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:17.970620 16550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 07:11:17.970829 16550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "b6820f423db64b6285a60f6e7f472344"
format_stamp: "Formatted at 2026-09-27 07:11:17 on dist-test-slave-8vqf"
server_key: "c584b1ed36be9521a2e00f368f1303f9"
server_key_iv: "a89904fb7ed7b977d36c631055610cf0"
server_key_version: "encryptionkey@0"
I20260927 07:11:17.971045 16550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:17.993335 16550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:17.999006 16550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:17.999352 16550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:18.020437 16550 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:43547
I20260927 07:11:18.020459 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:43547 every 8 connection(s)
I20260927 07:11:18.022037 16550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:18.025923 16617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:18.027737 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16550
I20260927 07:11:18.027918 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:18.028496 30785 external_mini_cluster.cc:1428] Setting key efae9bc71c94bf0b88ca251ca53929d3
I20260927 07:11:18.041419 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: Bootstrap starting.
Sep 27 07:11:18 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:18.045581 16617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:18.046691 16617 log.cc:824] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:18.050009 16617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: No bootstrap required, opened a new log
I20260927 07:11:18.053807 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:18.030410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.30.16.126:43547)
0927 07:11:18.031360 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:18.031381 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:18.031449 (+ 68us) server_negotiation.cc:407] Connection header received
0927 07:11:18.033154 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:18.033177 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:18.033673 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:18.034032 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:18.035936 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.037256 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.039210 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.039996 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.040155 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:18.045378 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:18.045387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:18.045403 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:18.045447 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:18.049073 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:18.049938 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:18.049945 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:18.049951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:18.050264 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:18.050795 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:18.050800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:18.050807 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:18.051393 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:18.051739 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:18.052281 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 07:11:18.052743 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":399,"threads_started":1}
I20260927 07:11:18.058329 16617 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } }
I20260927 07:11:18.058817 16617 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:18.058933 16617 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6820f423db64b6285a60f6e7f472344, State: Initialized, Role: FOLLOWER
I20260927 07:11:18.059469 16617 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } }
I20260927 07:11:18.059707 16617 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:18.059852 16617 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:18.060019 16617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:18.062170 16617 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } }
I20260927 07:11:18.062718 16617 leader_election.cc:302] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6820f423db64b6285a60f6e7f472344; no voters:
I20260927 07:11:18.063289 16617 leader_election.cc:288] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:18.063444 16622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:18.064134 16622 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [term 1 LEADER]: Becoming Leader. State: Replica: b6820f423db64b6285a60f6e7f472344, State: Running, Role: LEADER
I20260927 07:11:18.064664 16622 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } }
I20260927 07:11:18.065493 16617 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:18.067978 16624 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6820f423db64b6285a60f6e7f472344. Latest consensus state: current_term: 1 leader_uuid: "b6820f423db64b6285a60f6e7f472344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } } }
I20260927 07:11:18.068367 16624 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:18.069121 16623 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6820f423db64b6285a60f6e7f472344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6820f423db64b6285a60f6e7f472344" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 43547 } } }
I20260927 07:11:18.069532 16623 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:18.070080 16629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:18.075271 16629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:18.084268 16629 catalog_manager.cc:1347] Generated new cluster ID: 39eb16fc7d6b45a092b2365591479120
I20260927 07:11:18.084391 16629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:18.098112 16629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:18.099661 16629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:18.111017 16629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: Generated new TSK 0
I20260927 07:11:18.112171 16629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:11:18.177832 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:18.525393 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:18.525701 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:18.525874 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:18.533339 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:18.533475 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:18.533679 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:18.546525 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:18.547978 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:18.549412 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:18.563982 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:18.563962 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:18.564224 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:18.564896 16643 server_base.cc:1034] running on GCE node
I20260927 07:11:18.565632 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:18.566733 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:18.567948 16643 hybrid_clock.cc:648] HybridClock initialized: now 1790493078567900 us; error 63 us; skew 500 ppm
I20260927 07:11:18.568356 16643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:18 dist-test-slave-8vqf krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493078, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:18.571787 16643 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:11:18.573424 16643 webserver.cc:460] Webserver started at http://127.30.16.65:44531/ using document root <none> and password file <none>
I20260927 07:11:18.574124 16643 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:18.574226 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:18.574492 16643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:18.577790 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "a8fb030ce8684f79ac409a674c59cfc1"
format_stamp: "Formatted at 2026-09-27 07:11:18 on dist-test-slave-8vqf"
server_key: "679391ee629c9048ac16714a4ecce949"
server_key_iv: "8a37fc1d13ee4869576eb993d8432b55"
server_key_version: "encryptionkey@0"
I20260927 07:11:18.578589 16643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "a8fb030ce8684f79ac409a674c59cfc1"
format_stamp: "Formatted at 2026-09-27 07:11:18 on dist-test-slave-8vqf"
server_key: "679391ee629c9048ac16714a4ecce949"
server_key_iv: "8a37fc1d13ee4869576eb993d8432b55"
server_key_version: "encryptionkey@0"
I20260927 07:11:18.583905 16643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:11:18.587580 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:18.589007 16643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:18.589248 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "a8fb030ce8684f79ac409a674c59cfc1"
format_stamp: "Formatted at 2026-09-27 07:11:18 on dist-test-slave-8vqf"
server_key: "679391ee629c9048ac16714a4ecce949"
server_key_iv: "8a37fc1d13ee4869576eb993d8432b55"
server_key_version: "encryptionkey@0"
I20260927 07:11:18.589491 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:18.659956 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:18.664390 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:18.664733 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:18.665899 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:18.667518 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:18.667626 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:18.667795 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:18.667873 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:18.703698 16643 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:40923
I20260927 07:11:18.703728 16770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:40923 every 8 connection(s)
I20260927 07:11:18.705184 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:18.706902 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16643
I20260927 07:11:18.707055 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:18.707453 30785 external_mini_cluster.cc:1428] Setting key 4db9bbc448b6ba62863c5b6064e6c363
Sep 27 07:11:18 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493078, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:18.728856 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:18.709222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:60215 (local address 127.30.16.126:43547)
0927 07:11:18.709810 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:18.709818 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:18.710868 (+ 1050us) server_negotiation.cc:407] Connection header received
0927 07:11:18.712906 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:18.712912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:18.713014 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:18.713140 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:18.714954 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.715776 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.717529 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.717997 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.718142 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:18.722383 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:18.722389 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:18.722393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:18.722430 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:18.725266 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:18.726259 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:18.726264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:18.726267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:18.726396 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:18.726932 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:18.726935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:18.726938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:18.727223 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:18.727343 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:18.728361 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0927 07:11:18.728570 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
I20260927 07:11:18.730479 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:18.709770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43547 (local address 127.30.16.65:60215)
0927 07:11:18.710711 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:18.710754 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:18.712604 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:18.713420 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:18.713430 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:18.714027 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:18.714753 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:18.714776 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.715966 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.715971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:18.717372 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:18.717385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:18.718294 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:18.718302 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:18.718624 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:18.719388 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:18.719417 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:18.722097 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:18.725426 (+ 3329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:18.725438 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:18.725461 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:18.726118 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:18.726510 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:18.726517 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:18.726524 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:18.726812 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:18.727365 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:18.727376 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:18.727871 (+ 495us) client_negotiation.cc:769] Sending connection context
0927 07:11:18.728201 (+ 330us) client_negotiation.cc:240] Negotiation successful
0927 07:11:18.728839 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":383,"threads_started":1}
I20260927 07:11:18.732263 16771 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43547
I20260927 07:11:18.732613 16771 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
I20260927 07:11:18.733433 16771 heartbeater.cc:511] Master 127.30.16.126:43547 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
I20260927 07:11:18.735741 16582 ts_manager.cc:194] Registered new tserver with Master: a8fb030ce8684f79ac409a674c59cfc1 (127.30.16.65:40923)
I20260927 07:11:18.737210 16582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:60215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:11:18.763928 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:19.098758 16777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:19.099171 16777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:19.099328 16777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:19.107016 16777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:19.107174 16777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:19.107352 16777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:19.117595 16777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:19.119036 16777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:19.120434 16777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:19.133746 16783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:19.133867 16782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:19.133795 16785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:19.134526 16777 server_base.cc:1034] running on GCE node
I20260927 07:11:19.135181 16777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:19.136150 16777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:19.137382 16777 hybrid_clock.cc:648] HybridClock initialized: now 1790493079137323 us; error 55 us; skew 500 ppm
I20260927 07:11:19.137703 16777 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:19 dist-test-slave-8vqf krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493079, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:19.140914 16777 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:11:19.142341 16777 webserver.cc:460] Webserver started at http://127.30.16.66:43421/ using document root <none> and password file <none>
I20260927 07:11:19.143102 16777 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:19.143236 16777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:19.143553 16777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:19.146502 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "7916ab79e4274b8fa5cfe9f216ec301f"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "13d1ad6a07e953d2d10e6c500f984313"
server_key_iv: "fa52e196655a16355b8d7da858977e70"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.147373 16777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "7916ab79e4274b8fa5cfe9f216ec301f"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "13d1ad6a07e953d2d10e6c500f984313"
server_key_iv: "fa52e196655a16355b8d7da858977e70"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.152738 16777 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 07:11:19.156313 16792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.157735 16777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:19.157940 16777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "7916ab79e4274b8fa5cfe9f216ec301f"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "13d1ad6a07e953d2d10e6c500f984313"
server_key_iv: "fa52e196655a16355b8d7da858977e70"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.158157 16777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:19.178352 16777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:19.183060 16777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:19.183372 16777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:19.184357 16777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:19.185724 16777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:19.185835 16777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.185974 16777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:19.186043 16777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.220981 16777 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:46525
I20260927 07:11:19.221017 16904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:46525 every 8 connection(s)
I20260927 07:11:19.222399 16777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:19.228719 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16777
I20260927 07:11:19.228935 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:19.229405 30785 external_mini_cluster.cc:1428] Setting key 39fb87402dc379f8fb24467a25b26939
Sep 27 07:11:19 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493079, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:19.244038 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.225120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:57665 (local address 127.30.16.126:43547)
0927 07:11:19.225301 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.225306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.226543 (+ 1237us) server_negotiation.cc:407] Connection header received
0927 07:11:19.228463 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.228469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.228564 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.228674 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:19.230437 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.231449 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.233161 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.233628 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.233778 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.237524 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:19.237533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:19.237542 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:19.237591 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:19.240623 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.241617 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.241620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.241623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.241749 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.242249 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.242252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.242254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.242513 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:19.242691 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.243598 (+ 907us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.243785 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:11:19.245801 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.225616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43547 (local address 127.30.16.66:57665)
0927 07:11:19.226359 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:19.226410 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:19.228154 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:19.228965 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:19.228974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:19.229594 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:19.230227 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.230254 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.231655 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.231659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.233005 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.233019 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.233885 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.233893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.234138 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.234926 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:19.234957 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:19.237281 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:19.240772 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:19.240785 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:19.240807 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:19.241504 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:19.241856 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:19.241864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:19.241872 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:19.242141 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:19.242688 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:19.242700 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:19.243213 (+ 513us) client_negotiation.cc:769] Sending connection context
0927 07:11:19.243548 (+ 335us) client_negotiation.cc:240] Negotiation successful
0927 07:11:19.244188 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":278,"threads_started":1}
I20260927 07:11:19.247360 16905 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43547
I20260927 07:11:19.247671 16905 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:19.248430 16905 heartbeater.cc:511] Master 127.30.16.126:43547 requested a full tablet report, sending...
I20260927 07:11:19.249953 16582 ts_manager.cc:194] Registered new tserver with Master: 7916ab79e4274b8fa5cfe9f216ec301f (127.30.16.66:46525)
I20260927 07:11:19.250856 16582 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:57665
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:11:19.287073 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:19.622650 16910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:19.623023 16910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:19.623198 16910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:19.630620 16910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:19.630780 16910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:19.630966 16910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:19.640833 16910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:43547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:19.642271 16910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:19.643854 16910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:19.657331 16915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:19.657253 16918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:19.657290 16916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:19.658195 16910 server_base.cc:1034] running on GCE node
I20260927 07:11:19.658916 16910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:19.659926 16910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:19.661152 16910 hybrid_clock.cc:648] HybridClock initialized: now 1790493079661104 us; error 42 us; skew 500 ppm
I20260927 07:11:19.661486 16910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:19 dist-test-slave-8vqf krb5kdc[16537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493079, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:19.664801 16910 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:11:19.666433 16910 webserver.cc:460] Webserver started at http://127.30.16.67:46875/ using document root <none> and password file <none>
I20260927 07:11:19.667259 16910 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:19.667407 16910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:19.667733 16910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:19.670816 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "048b5e6c720de38e62a045b37d76bfe8"
server_key_iv: "b7122b380b4c5a4be35169b18aaa86f7"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.671667 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "048b5e6c720de38e62a045b37d76bfe8"
server_key_iv: "b7122b380b4c5a4be35169b18aaa86f7"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.677304 16910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 07:11:19.680989 16925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.682379 16910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:11:19.682616 16910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8"
format_stamp: "Formatted at 2026-09-27 07:11:19 on dist-test-slave-8vqf"
server_key: "048b5e6c720de38e62a045b37d76bfe8"
server_key_iv: "b7122b380b4c5a4be35169b18aaa86f7"
server_key_version: "encryptionkey@0"
I20260927 07:11:19.682826 16910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:19.700927 16910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:19.705370 16910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:19.705668 16910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:19.706735 16910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:19.708348 16910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:19.708451 16910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.708600 16910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:19.708668 16910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:19.740983 16771 heartbeater.cc:503] Master 127.30.16.126:43547 was elected leader, sending a full tablet report...
I20260927 07:11:19.742617 16910 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:33137
I20260927 07:11:19.742650 17037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:33137 every 8 connection(s)
I20260927 07:11:19.744081 16910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:19.751917 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 16910
I20260927 07:11:19.752102 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:19.752578 30785 external_mini_cluster.cc:1428] Setting key 2ea174465827c9a4488a6f99575c95c2
Sep 27 07:11:19 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493079, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:19.764731 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.746690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:35805 (local address 127.30.16.126:43547)
0927 07:11:19.747146 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.747150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.748186 (+ 1036us) server_negotiation.cc:407] Connection header received
0927 07:11:19.750047 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.750053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.750141 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.750263 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:19.751877 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.752862 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.754877 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.755391 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.755514 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.759071 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:19.759076 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:19.759080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:19.759115 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:19.761804 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.762543 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.762585 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.762589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.762723 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.763135 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.763139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.763141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.763420 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:19.763540 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.764278 (+ 738us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.764460 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":251,"threads_started":1}
I20260927 07:11:19.765796 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.747328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:43547 (local address 127.30.16.67:35805)
0927 07:11:19.748031 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:19.748069 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:19.749760 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:19.750518 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:19.750527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:19.751133 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:19.751706 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.751722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.753021 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.753026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.754723 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.754736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.755629 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.755636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.755870 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.756499 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:19.756527 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:19.758846 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:19.761933 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:19.761957 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:19.761976 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:19.762431 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:19.762819 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:19.762823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:19.762826 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:19.763020 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:19.763528 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:19.763534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:19.763914 (+ 380us) client_negotiation.cc:769] Sending connection context
0927 07:11:19.764176 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 07:11:19.764604 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":256,"threads_started":1}
I20260927 07:11:19.767364 17038 heartbeater.cc:347] Connected to a master server at 127.30.16.126:43547
I20260927 07:11:19.767685 17038 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:19.768500 17038 heartbeater.cc:511] Master 127.30.16.126:43547 requested a full tablet report, sending...
I20260927 07:11:19.769920 16581 ts_manager.cc:194] Registered new tserver with Master: 22dfcef0b4ce49ec8ccde92cba68c1a8 (127.30.16.67:33137)
I20260927 07:11:19.770901 16581 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:35805
I20260927 07:11:19.780352 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:19.797896 17040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.784328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.30.16.126:43547)
0927 07:11:19.784508 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.784516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.784927 (+ 411us) server_negotiation.cc:407] Connection header received
0927 07:11:19.785130 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.785134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.785209 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.785318 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:19.786905 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.787649 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.788985 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.789403 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.789532 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.792410 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:19.792415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:19.792418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:19.792454 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:19.795090 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.795878 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.795884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.795888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.796011 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:19.796448 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:19.796453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:19.796456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:19.796771 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:19.797045 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.797400 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.797595 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 07:11:19.801859 16581 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:19.808836 16581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:19.852675 17053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:40923 (local address 127.0.0.1:42942)
0927 07:11:19.837438 (+ 16354us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:19.837498 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:19.837772 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:19.844141 (+ 6369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:19.844150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:19.844265 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:19.845454 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.845474 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.847215 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.847219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.849417 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.849433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.851485 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.851492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.851744 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.851809 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 07:11:19.851932 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 07:11:19.852150 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16051,"mutex_wait_us":90,"spinlock_wait_cycles":120576,"thread_start_us":21451,"threads_started":1}
I20260927 07:11:19.852797 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41550 (local address 127.30.16.66:46525)
0927 07:11:19.842063 (+ 20162us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.842077 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.842485 (+ 408us) server_negotiation.cc:407] Connection header received
0927 07:11:19.843412 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.843419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.843693 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.843902 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:19.845477 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.846875 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.849589 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.851192 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.851322 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.851538 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.852277 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.852452 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19889,"thread_start_us":20192,"threads_started":1}
I20260927 07:11:19.853433 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42942 (local address 127.30.16.65:40923)
0927 07:11:19.843399 (+ 22042us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.843408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.843436 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:11:19.843539 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.843547 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.843837 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.844272 (+ 435us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:19.845608 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.847050 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.849589 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.851229 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.852460 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.852721 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.852801 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.853014 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21862,"thread_start_us":21652,"threads_started":1}
I20260927 07:11:19.853451 17051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46525 (local address 127.0.0.1:41550)
0927 07:11:19.842318 (+ 20618us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:19.842344 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:19.843161 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:19.844150 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:19.844153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:19.844202 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:19.845108 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.845124 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.847014 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.847020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.849383 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.849402 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.851687 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.851693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.851900 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.851931 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 07:11:19.852019 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 07:11:19.852163 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20422,"mutex_wait_us":61,"spinlock_wait_cycles":10368,"thread_start_us":17586,"threads_started":1}
I20260927 07:11:19.854202 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.30.16.67:33137)
0927 07:11:19.842959 (+ 21049us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:19.842972 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:19.843013 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:11:19.843684 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:19.843691 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:19.843896 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:19.844516 (+ 620us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:19.845773 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.848501 (+ 2728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.851056 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.852549 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.852673 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.853473 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:19.853798 (+ 325us) server_negotiation.cc:299] Negotiation successful
0927 07:11:19.853931 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20822,"thread_start_us":20674,"threads_started":1}
I20260927 07:11:19.854988 17054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:19.821333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:33137 (local address 127.0.0.1:45522)
0927 07:11:19.839662 (+ 18329us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:19.839689 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:19.839854 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:19.844141 (+ 4287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:19.844153 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:19.844253 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:19.845537 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.845557 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.848669 (+ 3112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.848674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.850857 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:19.850874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:19.853193 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:19.853199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:19.853402 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:19.853446 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 07:11:19.853560 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:11:19.854524 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18141,"mutex_wait_us":1,"spinlock_wait_cycles":75520,"thread_start_us":15623,"threads_started":1}
I20260927 07:11:19.857363 16706 tablet_service.cc:1467] Processing CreateTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 (DEFAULT_TABLE table=test-table [id=5dc610b6deff44b8b3a34b964add9198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:19.857622 16840 tablet_service.cc:1467] Processing CreateTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 (DEFAULT_TABLE table=test-table [id=5dc610b6deff44b8b3a34b964add9198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:19.858712 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1501304cc16649e39e6d03c0d9c386c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:19.859153 16840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1501304cc16649e39e6d03c0d9c386c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:19.859189 16973 tablet_service.cc:1467] Processing CreateTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 (DEFAULT_TABLE table=test-table [id=5dc610b6deff44b8b3a34b964add9198]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:19.860684 16973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1501304cc16649e39e6d03c0d9c386c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:19.870587 17057 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Bootstrap starting.
I20260927 07:11:19.873901 17057 tablet_bootstrap.cc:654] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:19.875638 17057 log.cc:824] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:19.878062 17058 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Bootstrap starting.
I20260927 07:11:19.878885 17057 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: No bootstrap required, opened a new log
I20260927 07:11:19.879170 17057 ts_tablet_manager.cc:1397] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260927 07:11:19.882134 17059 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Bootstrap starting.
I20260927 07:11:19.883386 17058 tablet_bootstrap.cc:654] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:19.884922 17058 log.cc:824] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:19.886201 17059 tablet_bootstrap.cc:654] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:19.887496 17059 log.cc:824] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:19.888484 17058 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: No bootstrap required, opened a new log
I20260927 07:11:19.888784 17058 ts_tablet_manager.cc:1397] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260927 07:11:19.890106 17057 raft_consensus.cc:348] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.890882 17057 raft_consensus.cc:374] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:19.891017 17057 raft_consensus.cc:729] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8fb030ce8684f79ac409a674c59cfc1, State: Initialized, Role: FOLLOWER
I20260927 07:11:19.891706 17057 consensus_queue.cc:260] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.894093 17059 tablet_bootstrap.cc:492] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: No bootstrap required, opened a new log
I20260927 07:11:19.894726 17059 ts_tablet_manager.cc:1397] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260927 07:11:19.898214 17058 raft_consensus.cc:348] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.898696 17058 raft_consensus.cc:374] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:19.898871 17058 raft_consensus.cc:729] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7916ab79e4274b8fa5cfe9f216ec301f, State: Initialized, Role: FOLLOWER
I20260927 07:11:19.899812 17058 consensus_queue.cc:260] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.900020 17057 ts_tablet_manager.cc:1428] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Time spent starting tablet: real 0.021s user 0.011s sys 0.008s
I20260927 07:11:19.901139 16905 heartbeater.cc:503] Master 127.30.16.126:43547 was elected leader, sending a full tablet report...
I20260927 07:11:19.901990 17058 ts_tablet_manager.cc:1428] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260927 07:11:19.905787 17059 raft_consensus.cc:348] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.906325 17059 raft_consensus.cc:374] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:19.906459 17059 raft_consensus.cc:729] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dfcef0b4ce49ec8ccde92cba68c1a8, State: Initialized, Role: FOLLOWER
I20260927 07:11:19.907045 17059 consensus_queue.cc:260] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:19.908073 17038 heartbeater.cc:503] Master 127.30.16.126:43547 was elected leader, sending a full tablet report...
I20260927 07:11:19.908670 17059 ts_tablet_manager.cc:1428] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260927 07:11:19.957888 16772 tablet.cc:2369] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:19.974525 16906 tablet.cc:2369] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:19.995723 17039 tablet.cc:2369] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:20.164172 17065 raft_consensus.cc:482] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:20.164503 17065 raft_consensus.cc:504] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:20.166313 17065 leader_election.cc:288] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8fb030ce8684f79ac409a674c59cfc1 (127.30.16.65:40923), 7916ab79e4274b8fa5cfe9f216ec301f (127.30.16.66:46525)
I20260927 07:11:20.172217 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.166720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:49977 (local address 127.30.16.65:40923)
0927 07:11:20.166960 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.166968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.167280 (+ 312us) server_negotiation.cc:407] Connection header received
0927 07:11:20.167398 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.167402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.167460 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.167559 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:20.167941 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.168662 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.170314 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.171338 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.171506 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.171582 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.171790 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.171948 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 07:11:20.172310 17041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.166623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:40923 (local address 127.30.16.67:49977)
0927 07:11:20.167051 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:20.167072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:20.167221 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:20.167579 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:20.167584 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:20.167610 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:20.167795 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:20.167805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.168802 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.168807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:20.170150 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:20.170161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.171499 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.171503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:20.171637 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.171662 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:11:20.171884 (+ 222us) client_negotiation.cc:240] Negotiation successful
0927 07:11:20.172075 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":39}
I20260927 07:11:20.173314 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.166768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:46525 (local address 127.30.16.67:52425)
0927 07:11:20.167542 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:20.167556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:20.167774 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:20.168087 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:20.168091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:20.168113 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:20.168437 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:20.168446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.169449 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.169455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:20.170873 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:20.170884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.172618 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.172624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:20.172797 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.172852 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:11:20.172945 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 07:11:20.173039 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":1,"spinlock_wait_cycles":24448,"thread_start_us":454,"threads_started":1}
I20260927 07:11:20.173408 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.167004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:52425 (local address 127.30.16.66:46525)
0927 07:11:20.167202 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.167208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.167649 (+ 441us) server_negotiation.cc:407] Connection header received
0927 07:11:20.167919 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.167924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.167983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.168075 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:20.168597 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.169303 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.171056 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.172330 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.172483 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.172574 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.173010 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.173163 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:20.173533 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1501304cc16649e39e6d03c0d9c386c6" candidate_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb030ce8684f79ac409a674c59cfc1" is_pre_election: true
I20260927 07:11:20.174163 16726 raft_consensus.cc:2454] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22dfcef0b4ce49ec8ccde92cba68c1a8 in term 0.
I20260927 07:11:20.174399 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1501304cc16649e39e6d03c0d9c386c6" candidate_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" is_pre_election: true
I20260927 07:11:20.174970 16860 raft_consensus.cc:2454] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22dfcef0b4ce49ec8ccde92cba68c1a8 in term 0.
I20260927 07:11:20.174952 16928 leader_election.cc:302] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22dfcef0b4ce49ec8ccde92cba68c1a8, a8fb030ce8684f79ac409a674c59cfc1; no voters:
I20260927 07:11:20.175467 17065 raft_consensus.cc:2781] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:20.175619 17065 raft_consensus.cc:482] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:20.175729 17065 raft_consensus.cc:3037] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:20.178063 17065 raft_consensus.cc:504] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:20.178853 17065 leader_election.cc:288] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [CANDIDATE]: Term 1 election: Requested vote from peers a8fb030ce8684f79ac409a674c59cfc1 (127.30.16.65:40923), 7916ab79e4274b8fa5cfe9f216ec301f (127.30.16.66:46525)
I20260927 07:11:20.179275 16860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1501304cc16649e39e6d03c0d9c386c6" candidate_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7916ab79e4274b8fa5cfe9f216ec301f"
I20260927 07:11:20.179360 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1501304cc16649e39e6d03c0d9c386c6" candidate_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8fb030ce8684f79ac409a674c59cfc1"
I20260927 07:11:20.179467 16860 raft_consensus.cc:3037] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:20.179628 16726 raft_consensus.cc:3037] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:20.181708 16860 raft_consensus.cc:2454] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dfcef0b4ce49ec8ccde92cba68c1a8 in term 1.
I20260927 07:11:20.182237 16726 raft_consensus.cc:2454] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22dfcef0b4ce49ec8ccde92cba68c1a8 in term 1.
I20260927 07:11:20.182276 16929 leader_election.cc:302] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22dfcef0b4ce49ec8ccde92cba68c1a8, 7916ab79e4274b8fa5cfe9f216ec301f; no voters:
I20260927 07:11:20.182638 17065 raft_consensus.cc:2781] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:20.183058 17065 raft_consensus.cc:686] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 22dfcef0b4ce49ec8ccde92cba68c1a8, State: Running, Role: LEADER
I20260927 07:11:20.183588 17065 consensus_queue.cc:237] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } }
I20260927 07:11:20.188429 16582 catalog_manager.cc:5495] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22dfcef0b4ce49ec8ccde92cba68c1a8 (127.30.16.67). New cstate: current_term: 1 leader_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22dfcef0b4ce49ec8ccde92cba68c1a8" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 33137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 } health_report { overall_health: UNKNOWN } } }
I20260927 07:11:20.225013 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.219401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.30.16.67:33137)
0927 07:11:20.219625 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.219632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.219750 (+ 118us) server_negotiation.cc:407] Connection header received
0927 07:11:20.219993 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.219997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.220085 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.220241 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:20.220667 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.221378 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.222817 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.223210 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.223339 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.223951 (+ 612us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:20.224435 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:20.224618 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.224740 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.224789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:11:20.233312 16726 raft_consensus.cc:1261] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 1 FOLLOWER]: Refusing update from remote peer 22dfcef0b4ce49ec8ccde92cba68c1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:20.233325 16860 raft_consensus.cc:1261] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 1 FOLLOWER]: Refusing update from remote peer 22dfcef0b4ce49ec8ccde92cba68c1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:20.234645 17067 consensus_queue.cc:1035] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8fb030ce8684f79ac409a674c59cfc1" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 40923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:20.235033 17065 consensus_queue.cc:1035] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7916ab79e4274b8fa5cfe9f216ec301f" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:20.251672 17070 mvcc.cc:205] Tried to move back new op lower bound from 7333859656625311744 to 7333859656436203520. Current Snapshot: MvccSnapshot[applied={T|T < 7333859656625311744 or (T in {7333859656625311744})}]
I20260927 07:11:20.262437 16582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:20.262823 16582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:20.266484 16582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6820f423db64b6285a60f6e7f472344: Sending DeleteTablet for 3 replicas of tablet 1501304cc16649e39e6d03c0d9c386c6
I20260927 07:11:20.267645 16973 tablet_service.cc:1514] Processing DeleteTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:45522
I20260927 07:11:20.267712 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:42942
I20260927 07:11:20.268671 17079 tablet_replica.cc:330] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: stopping tablet replica
I20260927 07:11:20.269084 16840 tablet_service.cc:1514] Processing DeleteTablet for tablet 1501304cc16649e39e6d03c0d9c386c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:20 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:41550
I20260927 07:11:20.269207 17079 raft_consensus.cc:2229] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:20.270085 17079 raft_consensus.cc:2258] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:20.271906 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16643
I20260927 07:11:20.272555 17079 ts_tablet_manager.cc:1905] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:20.281529 17079 ts_tablet_manager.cc:1918] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:20.281890 17079 log.cc:1196] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/1501304cc16649e39e6d03c0d9c386c6
I20260927 07:11:20.282641 17079 ts_tablet_manager.cc:1939] T 1501304cc16649e39e6d03c0d9c386c6 P 22dfcef0b4ce49ec8ccde92cba68c1a8: Deleting consensus metadata
I20260927 07:11:20.284957 16566 catalog_manager.cc:4841] TS 22dfcef0b4ce49ec8ccde92cba68c1a8 (127.30.16.67:33137): tablet 1501304cc16649e39e6d03c0d9c386c6 (table test-table [id=5dc610b6deff44b8b3a34b964add9198]) successfully deleted
I20260927 07:11:20.288030 17081 tablet_replica.cc:330] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: stopping tablet replica
I20260927 07:11:20.288622 17081 raft_consensus.cc:2229] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:20.289345 17081 raft_consensus.cc:2258] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:20.290303 17080 tablet_replica.cc:330] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: stopping tablet replica
I20260927 07:11:20.290802 17080 raft_consensus.cc:2229] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:20.291324 17080 raft_consensus.cc:2258] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:20.292438 17081 ts_tablet_manager.cc:1905] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:20.293457 17080 ts_tablet_manager.cc:1905] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 07:11:20 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:11:20.299888 17081 ts_tablet_manager.cc:1918] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:20.300088 17081 log.cc:1196] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/1501304cc16649e39e6d03c0d9c386c6
I20260927 07:11:20.300669 17081 ts_tablet_manager.cc:1939] T 1501304cc16649e39e6d03c0d9c386c6 P 7916ab79e4274b8fa5cfe9f216ec301f: Deleting consensus metadata
I20260927 07:11:20.301596 17080 ts_tablet_manager.cc:1918] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:20.301836 17080 log.cc:1196] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/1501304cc16649e39e6d03c0d9c386c6
I20260927 07:11:20.302418 17080 ts_tablet_manager.cc:1939] T 1501304cc16649e39e6d03c0d9c386c6 P a8fb030ce8684f79ac409a674c59cfc1: Deleting consensus metadata
I20260927 07:11:20.302762 16569 catalog_manager.cc:4841] TS 7916ab79e4274b8fa5cfe9f216ec301f (127.30.16.66:46525): tablet 1501304cc16649e39e6d03c0d9c386c6 (table test-table [id=5dc610b6deff44b8b3a34b964add9198]) successfully deleted
I20260927 07:11:20.304418 16568 catalog_manager.cc:4841] TS a8fb030ce8684f79ac409a674c59cfc1 (127.30.16.65:40923): tablet 1501304cc16649e39e6d03c0d9c386c6 (table test-table [id=5dc610b6deff44b8b3a34b964add9198]) successfully deleted
I20260927 07:11:20.307765 17052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.274079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.30.16.65:40923)
0927 07:11:20.274312 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.274319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.287287 (+ 12968us) server_negotiation.cc:407] Connection header received
0927 07:11:20.287504 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.287513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.287651 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.287767 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:20.289804 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.291086 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.293162 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.293875 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.293988 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.300098 (+ 6110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:20.300119 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:20.300142 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:20.300209 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:20.304676 (+ 4467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.305640 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.305647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.305650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.305790 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.306189 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.306199 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.306201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.306631 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:20.306875 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.307285 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.307454 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:11:20.308492 16766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:20.401654 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16643
I20260927 07:11:20.419979 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16777
Sep 27 07:11:20 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:11:20.436040 17055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.420845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.30.16.66:46525)
0927 07:11:20.421039 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.421045 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.421068 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:11:20.421322 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.421328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.421405 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.421524 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:20.423025 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.423818 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.425111 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.425536 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.425677 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.430033 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:20.430049 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:20.430066 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:20.430108 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:20.433023 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.433781 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.433790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.433792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.433939 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.434330 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.434339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.434342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.434790 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:20.435094 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.435429 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.435665 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:11:20.436709 16900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:20.529057 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16777
I20260927 07:11:20.550486 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16910
Sep 27 07:11:20 dist-test-slave-8vqf krb5kdc[16537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:11:20.568715 17056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:20.551448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45546 (local address 127.30.16.67:33137)
0927 07:11:20.551648 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:20.551654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:20.551686 (+ 32us) server_negotiation.cc:407] Connection header received
0927 07:11:20.551757 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:20.551762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:20.551842 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:20.551962 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:20.553257 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.554528 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.555864 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:20.556544 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:20.556694 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:20.561407 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:20.561423 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:20.561445 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:20.561510 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:20.565563 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.566364 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.566373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.566377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.566612 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:20.566989 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:20.566994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:20.566998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:20.567491 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:20.567866 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:20.568027 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 07:11:20.568260 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:11:20.569607 17033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:20.696897 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16910
I20260927 07:11:20.717360 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 16550
I20260927 07:11:20.718779 16612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:20.831777 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 16550
2026-09-27T07:11:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T07:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:20Z Disabled control of system clock
I20260927 07:11:20.913280 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33149
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:21.258266 17090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:21.258626 17090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:21.258728 17090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:21.266850 17090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:21.266969 17090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:21.267007 17090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:21.267064 17090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:21.267115 17090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:21.276748 17090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:21.278136 17090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:21.280119 17090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:21.292546 17095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:21.292515 17096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:21.292779 17098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:21.293758 17090 server_base.cc:1034] running on GCE node
I20260927 07:11:21.294708 17090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:21.296283 17090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:21.297670 17090 hybrid_clock.cc:648] HybridClock initialized: now 1790493081297615 us; error 49 us; skew 500 ppm
I20260927 07:11:21.298082 17090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:21.301970 17090 webserver.cc:460] Webserver started at http://127.30.16.126:43511/ using document root <none> and password file <none>
I20260927 07:11:21.302886 17090 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:21.303025 17090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:21.303370 17090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:21.306519 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "ba928e5c0a754db1862b46441358cd2f"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "5205027f73d082414cb0dceb9273085a"
server_key_iv: "8e1516255b34361966db954062c0513a"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.307474 17090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "ba928e5c0a754db1862b46441358cd2f"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "5205027f73d082414cb0dceb9273085a"
server_key_iv: "8e1516255b34361966db954062c0513a"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.313599 17090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260927 07:11:21.317559 17104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:21.319262 17090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 07:11:21.319482 17090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "ba928e5c0a754db1862b46441358cd2f"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "5205027f73d082414cb0dceb9273085a"
server_key_iv: "8e1516255b34361966db954062c0513a"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.319742 17090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:21.335062 17090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:21.335984 17090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:21.336287 17090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:21.356796 17090 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33149
I20260927 07:11:21.356828 17155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33149 every 8 connection(s)
I20260927 07:11:21.358261 17090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:21.362308 17156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:21.368420 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17090
I20260927 07:11:21.368613 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:21.369218 30785 external_mini_cluster.cc:1428] Setting key 782f285559faa86b669af6c1b8592270
I20260927 07:11:21.374130 17156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: Bootstrap starting.
I20260927 07:11:21.377831 17156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:21.379066 17156 log.cc:824] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:21.382133 17156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: No bootstrap required, opened a new log
I20260927 07:11:21.384585 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:21.371424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.30.16.126:33149)
0927 07:11:21.372547 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:21.372559 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:21.372602 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:11:21.374648 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:21.374674 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:21.375191 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:21.375602 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:21.376195 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.378013 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.379627 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.380444 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.380646 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:21.381847 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:21.381856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:21.381870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:21.382753 (+ 883us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:21.382854 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:21.382878 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:21.383024 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:21.383262 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 07:11:21.383590 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":358,"threads_started":1}
I20260927 07:11:21.391027 17156 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } }
I20260927 07:11:21.391451 17156 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:21.391564 17156 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba928e5c0a754db1862b46441358cd2f, State: Initialized, Role: FOLLOWER
I20260927 07:11:21.392123 17156 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } }
I20260927 07:11:21.392373 17156 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:21.392513 17156 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:21.392681 17156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:21.395311 17156 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } }
I20260927 07:11:21.395826 17156 leader_election.cc:302] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba928e5c0a754db1862b46441358cd2f; no voters:
I20260927 07:11:21.396431 17156 leader_election.cc:288] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:21.396592 17161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:21.397343 17161 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [term 1 LEADER]: Becoming Leader. State: Replica: ba928e5c0a754db1862b46441358cd2f, State: Running, Role: LEADER
I20260927 07:11:21.397897 17161 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } }
I20260927 07:11:21.398653 17156 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:21.400717 17163 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba928e5c0a754db1862b46441358cd2f. Latest consensus state: current_term: 1 leader_uuid: "ba928e5c0a754db1862b46441358cd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } } }
I20260927 07:11:21.401142 17163 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:21.401192 17162 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba928e5c0a754db1862b46441358cd2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba928e5c0a754db1862b46441358cd2f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33149 } } }
I20260927 07:11:21.401510 17162 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:21.402273 17168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:21.409080 17168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:21.418915 17168 catalog_manager.cc:1347] Generated new cluster ID: 69504684fe2942a1a41c187bacc6cc6d
I20260927 07:11:21.419071 17168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:21.428695 17168 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:21.430019 17168 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:21.438105 17168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: Generated new TSK 0
I20260927 07:11:21.439220 17168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:21.446154 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--builtin_ntp_servers=127.30.16.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:21.803799 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:21.804140 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:21.804291 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:21.812155 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:21.812330 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:21.812502 17180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:21.822351 17180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:21.823992 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:21.825939 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:21.839685 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:21.839721 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:21.839917 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:21.840641 17180 server_base.cc:1034] running on GCE node
I20260927 07:11:21.841362 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:21.842375 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:21.843641 17180 hybrid_clock.cc:648] HybridClock initialized: now 1790493081843570 us; error 61 us; skew 500 ppm
I20260927 07:11:21.844003 17180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:21.846288 17180 webserver.cc:460] Webserver started at http://127.30.16.65:42945/ using document root <none> and password file <none>
I20260927 07:11:21.847086 17180 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:21.847218 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:21.847538 17180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:21.850521 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "e7c28afcaa6049609ae6ad74fd3f3554"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "c48044e13da7e7f564835671c21e2141"
server_key_iv: "4a9efdcd7420d8d008bfc6ac2a135d77"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.851426 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c28afcaa6049609ae6ad74fd3f3554"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "c48044e13da7e7f564835671c21e2141"
server_key_iv: "4a9efdcd7420d8d008bfc6ac2a135d77"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.856968 17180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:21.860525 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:21.861879 17180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 07:11:21.862080 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "e7c28afcaa6049609ae6ad74fd3f3554"
format_stamp: "Formatted at 2026-09-27 07:11:21 on dist-test-slave-8vqf"
server_key: "c48044e13da7e7f564835671c21e2141"
server_key_iv: "4a9efdcd7420d8d008bfc6ac2a135d77"
server_key_version: "encryptionkey@0"
I20260927 07:11:21.862336 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:21.879084 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:21.880070 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:21.880394 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:21.881426 17180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:21.883141 17180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:21.883277 17180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:21.883404 17180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:21.883488 17180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:21.917930 17180 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:44849
I20260927 07:11:21.917984 17306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:44849 every 8 connection(s)
I20260927 07:11:21.919437 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:21.923511 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17180
I20260927 07:11:21.923714 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:21.924264 30785 external_mini_cluster.cc:1428] Setting key eeaa6ecb178dcddf4ea97c5be8340b6b
I20260927 07:11:21.927363 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--builtin_ntp_servers=127.30.16.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:21.934657 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:21.922418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:44421 (local address 127.30.16.126:33149)
0927 07:11:21.922931 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:21.922936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:21.924443 (+ 1507us) server_negotiation.cc:407] Connection header received
0927 07:11:21.926417 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:21.926421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:21.926487 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:21.926639 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:21.928138 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.928894 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.931497 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.931904 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.932008 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:21.933796 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:21.933800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:21.933804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:21.933827 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:21.933872 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:21.933885 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:21.933959 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:21.934232 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 07:11:21.934329 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":289,"threads_started":1}
I20260927 07:11:21.936231 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:21.923079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33149 (local address 127.30.16.65:44421)
0927 07:11:21.924236 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:21.924299 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:21.926090 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:21.926906 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:21.926925 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:21.927062 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:21.927959 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:21.927988 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.929051 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.929059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:21.931337 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:21.931356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:21.932152 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:21.932161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:21.932526 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:21.933395 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:21.933400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:21.933461 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:21.933580 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:21.933986 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:21.934001 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:21.934013 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:11:21.934218 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 07:11:21.934744 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":319,"threads_started":1}
I20260927 07:11:21.938045 17307 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33149
I20260927 07:11:21.938421 17307 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:21.939289 17307 heartbeater.cc:511] Master 127.30.16.126:33149 requested a full tablet report, sending...
I20260927 07:11:21.941881 17121 ts_manager.cc:194] Registered new tserver with Master: e7c28afcaa6049609ae6ad74fd3f3554 (127.30.16.65:44849)
I20260927 07:11:21.944386 17121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.65:44421
W20260927 07:11:22.284427 17311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:22.284770 17311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:22.284860 17311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:22.292251 17311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:22.292364 17311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:22.292474 17311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:22.302063 17311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:22.303555 17311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:22.305814 17311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:22.320067 17317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:22.320230 17319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:22.320148 17316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:22.320974 17311 server_base.cc:1034] running on GCE node
I20260927 07:11:22.321647 17311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:22.322775 17311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:22.324035 17311 hybrid_clock.cc:648] HybridClock initialized: now 1790493082323989 us; error 38 us; skew 500 ppm
I20260927 07:11:22.324476 17311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:22.327107 17311 webserver.cc:460] Webserver started at http://127.30.16.66:40941/ using document root <none> and password file <none>
I20260927 07:11:22.327888 17311 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:22.328027 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:22.328361 17311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:22.331558 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "588f907cea3c42628a9d6b14d56c53e9"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "a867a827e234488225b6a7aa350da3ab"
server_key_iv: "f7761faac373561bab24a6504f1ac6b8"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.332383 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "588f907cea3c42628a9d6b14d56c53e9"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "a867a827e234488225b6a7aa350da3ab"
server_key_iv: "f7761faac373561bab24a6504f1ac6b8"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.337903 17311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:11:22.341702 17325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.343288 17311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:22.343487 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "588f907cea3c42628a9d6b14d56c53e9"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "a867a827e234488225b6a7aa350da3ab"
server_key_iv: "f7761faac373561bab24a6504f1ac6b8"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.343745 17311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:22.360431 17311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:22.361337 17311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:22.361675 17311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:22.362785 17311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:22.364439 17311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:22.364563 17311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.364706 17311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:22.364787 17311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.399838 17311 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:39735
I20260927 07:11:22.399887 17437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:39735 every 8 connection(s)
I20260927 07:11:22.401300 17311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:22.408288 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17311
I20260927 07:11:22.408501 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:22.409193 30785 external_mini_cluster.cc:1428] Setting key 824d820dc81e62a80f9c8d801f278981
I20260927 07:11:22.412803 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--builtin_ntp_servers=127.30.16.84:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:22.417539 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.404332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:53217 (local address 127.30.16.126:33149)
0927 07:11:22.404516 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.404521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.406331 (+ 1810us) server_negotiation.cc:407] Connection header received
0927 07:11:22.408765 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.408770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.408858 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.408956 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:22.410536 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.411574 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.414385 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.414850 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.414952 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.416755 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:22.416760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:22.416763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:22.416784 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:22.416816 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:22.416845 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:22.416917 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.417212 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.417298 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:11:22.419412 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.405013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33149 (local address 127.30.16.66:53217)
0927 07:11:22.406071 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:22.406176 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:22.408324 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:22.409294 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:22.409311 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:22.409453 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:22.410325 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.410353 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.411747 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.411753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.414219 (+ 2466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.414240 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.415114 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.415125 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.415502 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.416362 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:22.416368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:22.416431 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:22.416550 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:22.416950 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:22.416964 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:22.416976 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:11:22.417206 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 07:11:22.417699 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":368,"threads_started":1}
I20260927 07:11:22.421057 17438 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33149
I20260927 07:11:22.421463 17438 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:22.422364 17438 heartbeater.cc:511] Master 127.30.16.126:33149 requested a full tablet report, sending...
I20260927 07:11:22.424216 17121 ts_manager.cc:194] Registered new tserver with Master: 588f907cea3c42628a9d6b14d56c53e9 (127.30.16.66:39735)
I20260927 07:11:22.425561 17121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.66:53217
W20260927 07:11:22.767663 17441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:22.767969 17441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:22.768059 17441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:22.775605 17441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:22.775722 17441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:22.775835 17441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:22.785785 17441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:33149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:22.787182 17441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:22.788992 17441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:22.802901 17447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:22.803169 17449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:22.802860 17446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:22.803699 17441 server_base.cc:1034] running on GCE node
I20260927 07:11:22.804420 17441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:22.805456 17441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:22.806646 17441 hybrid_clock.cc:648] HybridClock initialized: now 1790493082806584 us; error 54 us; skew 500 ppm
I20260927 07:11:22.807003 17441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:22.809278 17441 webserver.cc:460] Webserver started at http://127.30.16.67:40985/ using document root <none> and password file <none>
I20260927 07:11:22.810029 17441 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:22.810158 17441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:22.810498 17441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:22.813445 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "742f567e91cb40d792a0bc2506fe901c"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "5b0d2c4c692084484c6d67767e005520"
server_key_iv: "99cd653741497c8e13d62bb89b137f4f"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.814268 17441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "742f567e91cb40d792a0bc2506fe901c"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "5b0d2c4c692084484c6d67767e005520"
server_key_iv: "99cd653741497c8e13d62bb89b137f4f"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.819658 17441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:11:22.823451 17455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.824971 17441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:22.825114 17441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "742f567e91cb40d792a0bc2506fe901c"
format_stamp: "Formatted at 2026-09-27 07:11:22 on dist-test-slave-8vqf"
server_key: "5b0d2c4c692084484c6d67767e005520"
server_key_iv: "99cd653741497c8e13d62bb89b137f4f"
server_key_version: "encryptionkey@0"
I20260927 07:11:22.825299 17441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:22.837901 17441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:22.838779 17441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:22.839040 17441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:22.839975 17441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:22.841610 17441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:22.841698 17441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.841784 17441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:22.841840 17441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:22.875907 17441 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:45035
I20260927 07:11:22.875947 17567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:45035 every 8 connection(s)
I20260927 07:11:22.877426 17441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:22.881964 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17441
I20260927 07:11:22.882233 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:22.882740 30785 external_mini_cluster.cc:1428] Setting key 71270666430aae6266474d5c542a7f0a
I20260927 07:11:22.890208 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.880112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:58031 (local address 127.30.16.126:33149)
0927 07:11:22.880283 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.880287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.881673 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 07:11:22.883847 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.883853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.883949 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.884056 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:22.885180 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.885928 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.887586 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.887977 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.888113 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.889414 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:22.889419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:22.889422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:22.889441 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:22.889471 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:22.889483 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:22.889575 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.889874 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.889956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:11:22.891363 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.880644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:33149 (local address 127.30.16.67:58031)
0927 07:11:22.881517 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:22.881553 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:22.883572 (+ 2019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:22.884293 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:22.884302 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:22.884409 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:22.885017 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.885034 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.886067 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.886077 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.887440 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.887452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.888200 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.888207 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.888449 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.889095 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:22.889097 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:22.889142 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:22.889220 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:22.889573 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:22.889584 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:22.889594 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 07:11:22.889850 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 07:11:22.890189 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
I20260927 07:11:22.892793 17568 heartbeater.cc:347] Connected to a master server at 127.30.16.126:33149
I20260927 07:11:22.893090 17568 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:22.893823 17568 heartbeater.cc:511] Master 127.30.16.126:33149 requested a full tablet report, sending...
I20260927 07:11:22.895226 17121 ts_manager.cc:194] Registered new tserver with Master: 742f567e91cb40d792a0bc2506fe901c (127.30.16.67:45035)
I20260927 07:11:22.896178 17121 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.67:58031
I20260927 07:11:22.897997 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:22.907758 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.902206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.30.16.126:33149)
0927 07:11:22.902411 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.902415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.902857 (+ 442us) server_negotiation.cc:407] Connection header received
0927 07:11:22.903028 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.903032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.903105 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.903242 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:22.903642 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.904406 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.905676 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.906078 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.906166 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.907202 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:22.907207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:22.907210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:22.907227 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:22.907260 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:22.907271 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:22.907384 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.907498 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.907542 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 07:11:22.912001 17120 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:22.919107 17120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:22.947981 17307 heartbeater.cc:503] Master 127.30.16.126:33149 was elected leader, sending a full tablet report...
I20260927 07:11:22.957839 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.933538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.30.16.66:39735)
0927 07:11:22.947933 (+ 14395us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.947939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.947973 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:11:22.948100 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.948112 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.948406 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.948653 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:22.949602 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.951222 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.954222 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.956077 (+ 1855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.956237 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.956525 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.957251 (+ 726us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.957450 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14249,"thread_start_us":14934,"threads_started":1}
I20260927 07:11:22.958647 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.932867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:39735 (local address 127.0.0.1:58188)
0927 07:11:22.947218 (+ 14351us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:22.947246 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:22.947403 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:22.948762 (+ 1359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:22.948770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:22.948802 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:22.949365 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.949381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.951411 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.951416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.954014 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.954028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.956682 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.956690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.956945 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.957016 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 07:11:22.957172 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 07:11:22.957397 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14138,"mutex_wait_us":3,"thread_start_us":17360,"threads_started":1}
I20260927 07:11:22.958891 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.932040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:44849 (local address 127.0.0.1:35268)
0927 07:11:22.933727 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:22.933766 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:22.933945 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:22.951836 (+ 17891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:22.951842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:22.951868 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:22.952294 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.952306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.954371 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.954377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.956241 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.956258 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.957963 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.957973 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.958149 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.958186 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:11:22.958294 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 07:11:22.958446 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1445,"thread_start_us":1312,"threads_started":1}
I20260927 07:11:22.958942 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.932368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.30.16.65:44849)
0927 07:11:22.951175 (+ 18807us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.951184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.951213 (+ 29us) server_negotiation.cc:407] Connection header received
0927 07:11:22.951329 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.951336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.951597 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.952486 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:22.952733 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.954243 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.956458 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.957629 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.957931 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.958211 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.958355 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.958509 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18638,"thread_start_us":19454,"threads_started":1}
I20260927 07:11:22.961551 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.934181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:45035 (local address 127.0.0.1:52842)
0927 07:11:22.950520 (+ 16339us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:22.950541 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:22.950689 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:22.951422 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:22.951430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:22.951469 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:22.953254 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.953265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.955263 (+ 1998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.955271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.957714 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:22.957729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.960740 (+ 3011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.960746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:22.960931 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.960964 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 07:11:22.961063 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 07:11:22.961218 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16203,"mutex_wait_us":1378,"thread_start_us":13303,"threads_started":1}
I20260927 07:11:22.961746 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:22.934542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.30.16.67:45035)
0927 07:11:22.947711 (+ 13169us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:22.947721 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:22.950665 (+ 2944us) server_negotiation.cc:407] Connection header received
0927 07:11:22.950834 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:22.950844 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:22.951160 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:22.951565 (+ 405us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:22.953473 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.954973 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.958012 (+ 3039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:22.960532 (+ 2520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:22.960658 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:22.960871 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:22.961180 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 07:11:22.961380 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12927,"thread_start_us":12705,"threads_started":1}
I20260927 07:11:22.963405 17242 tablet_service.cc:1467] Processing CreateTablet for tablet 686efcf61a91443cae0d8602dfcce52a (DEFAULT_TABLE table=test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:22.964056 17373 tablet_service.cc:1467] Processing CreateTablet for tablet 686efcf61a91443cae0d8602dfcce52a (DEFAULT_TABLE table=test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:22.964890 17242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686efcf61a91443cae0d8602dfcce52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:22.965473 17373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686efcf61a91443cae0d8602dfcce52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:22.965951 17503 tablet_service.cc:1467] Processing CreateTablet for tablet 686efcf61a91443cae0d8602dfcce52a (DEFAULT_TABLE table=test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:22.967429 17503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686efcf61a91443cae0d8602dfcce52a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:22.976958 17586 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Bootstrap starting.
I20260927 07:11:22.980358 17586 tablet_bootstrap.cc:654] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:22.981525 17586 log.cc:824] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:22.982024 17587 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Bootstrap starting.
I20260927 07:11:22.984177 17589 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Bootstrap starting.
I20260927 07:11:22.986076 17587 tablet_bootstrap.cc:654] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:22.986970 17586 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: No bootstrap required, opened a new log
I20260927 07:11:22.987262 17586 ts_tablet_manager.cc:1397] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260927 07:11:22.987409 17587 log.cc:824] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:22.987597 17589 tablet_bootstrap.cc:654] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:22.988956 17589 log.cc:824] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:22.990692 17587 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: No bootstrap required, opened a new log
I20260927 07:11:22.991024 17587 ts_tablet_manager.cc:1397] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260927 07:11:22.991942 17589 tablet_bootstrap.cc:492] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: No bootstrap required, opened a new log
I20260927 07:11:22.992211 17589 ts_tablet_manager.cc:1397] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 07:11:22.999293 17586 raft_consensus.cc:348] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:22.999846 17586 raft_consensus.cc:374] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:22.999956 17586 raft_consensus.cc:729] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c28afcaa6049609ae6ad74fd3f3554, State: Initialized, Role: FOLLOWER
I20260927 07:11:23.000713 17586 consensus_queue.cc:260] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.000955 17587 raft_consensus.cc:348] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.001461 17587 raft_consensus.cc:374] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:23.001574 17587 raft_consensus.cc:729] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 588f907cea3c42628a9d6b14d56c53e9, State: Initialized, Role: FOLLOWER
I20260927 07:11:23.002264 17587 consensus_queue.cc:260] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.002876 17586 ts_tablet_manager.cc:1428] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260927 07:11:23.002866 17589 raft_consensus.cc:348] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.003575 17589 raft_consensus.cc:374] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:23.003714 17589 raft_consensus.cc:729] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742f567e91cb40d792a0bc2506fe901c, State: Initialized, Role: FOLLOWER
I20260927 07:11:23.004020 17438 heartbeater.cc:503] Master 127.30.16.126:33149 was elected leader, sending a full tablet report...
I20260927 07:11:23.004599 17587 ts_tablet_manager.cc:1428] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260927 07:11:23.004575 17589 consensus_queue.cc:260] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.005851 17568 heartbeater.cc:503] Master 127.30.16.126:33149 was elected leader, sending a full tablet report...
I20260927 07:11:23.006579 17589 ts_tablet_manager.cc:1428] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260927 07:11:23.129446 17569 tablet.cc:2369] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:23.153671 17439 tablet.cc:2369] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:23.172122 17308 tablet.cc:2369] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:23.251255 17593 raft_consensus.cc:482] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:23.251609 17593 raft_consensus.cc:504] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.253285 17593 leader_election.cc:288] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c28afcaa6049609ae6ad74fd3f3554 (127.30.16.65:44849), 742f567e91cb40d792a0bc2506fe901c (127.30.16.67:45035)
I20260927 07:11:23.260136 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.253709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:34157 (local address 127.30.16.65:44849)
0927 07:11:23.253934 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.253943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.254480 (+ 537us) server_negotiation.cc:407] Connection header received
0927 07:11:23.254852 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.254858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.255030 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.255134 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:23.255723 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.256415 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.258385 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.259204 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.259393 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.259472 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.259756 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.259881 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:11:23.260224 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.254702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:33501 (local address 127.30.16.67:45035)
0927 07:11:23.254913 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.254919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.255239 (+ 320us) server_negotiation.cc:407] Connection header received
0927 07:11:23.255433 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.255442 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.255498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.255587 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:23.256099 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.256800 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.258576 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.259323 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.259496 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.259564 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.259828 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.259952 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:11:23.260387 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.253711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:44849 (local address 127.30.16.66:34157)
0927 07:11:23.254364 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:23.254380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:23.254584 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:23.255168 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:23.255173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:23.255202 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:23.255585 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:23.255593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.256548 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.256554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:23.258197 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:23.258208 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.259393 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.259399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:23.259585 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.259608 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 07:11:23.259714 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 07:11:23.259853 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":67,"spinlock_wait_cycles":26368,"thread_start_us":370,"threads_started":1}
I20260927 07:11:23.260690 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.254499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:45035 (local address 127.30.16.66:33501)
0927 07:11:23.255157 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:23.255171 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:23.255298 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:23.255607 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:23.255611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:23.255633 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:23.255962 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:23.255970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.256944 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.256948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:23.258383 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:23.258394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.259516 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.259522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:23.259682 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.259704 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 07:11:23.259802 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 07:11:23.259933 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":17,"thread_start_us":319,"threads_started":1}
I20260927 07:11:23.261521 17262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686efcf61a91443cae0d8602dfcce52a" candidate_uuid: "588f907cea3c42628a9d6b14d56c53e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" is_pre_election: true
I20260927 07:11:23.261652 17523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686efcf61a91443cae0d8602dfcce52a" candidate_uuid: "588f907cea3c42628a9d6b14d56c53e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742f567e91cb40d792a0bc2506fe901c" is_pre_election: true
I20260927 07:11:23.262159 17523 raft_consensus.cc:2454] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588f907cea3c42628a9d6b14d56c53e9 in term 0.
I20260927 07:11:23.262159 17262 raft_consensus.cc:2454] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 588f907cea3c42628a9d6b14d56c53e9 in term 0.
I20260927 07:11:23.262868 17329 leader_election.cc:302] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588f907cea3c42628a9d6b14d56c53e9, e7c28afcaa6049609ae6ad74fd3f3554; no voters:
I20260927 07:11:23.263223 17593 raft_consensus.cc:2781] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:23.263376 17593 raft_consensus.cc:482] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:23.263484 17593 raft_consensus.cc:3037] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:23.265626 17593 raft_consensus.cc:504] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.266271 17593 leader_election.cc:288] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [CANDIDATE]: Term 1 election: Requested vote from peers e7c28afcaa6049609ae6ad74fd3f3554 (127.30.16.65:44849), 742f567e91cb40d792a0bc2506fe901c (127.30.16.67:45035)
I20260927 07:11:23.266748 17262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686efcf61a91443cae0d8602dfcce52a" candidate_uuid: "588f907cea3c42628a9d6b14d56c53e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c28afcaa6049609ae6ad74fd3f3554"
I20260927 07:11:23.266844 17523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686efcf61a91443cae0d8602dfcce52a" candidate_uuid: "588f907cea3c42628a9d6b14d56c53e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "742f567e91cb40d792a0bc2506fe901c"
I20260927 07:11:23.266956 17262 raft_consensus.cc:3037] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:23.267099 17523 raft_consensus.cc:3037] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:23.269327 17262 raft_consensus.cc:2454] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588f907cea3c42628a9d6b14d56c53e9 in term 1.
I20260927 07:11:23.269493 17523 raft_consensus.cc:2454] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 588f907cea3c42628a9d6b14d56c53e9 in term 1.
I20260927 07:11:23.269775 17329 leader_election.cc:302] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 588f907cea3c42628a9d6b14d56c53e9, e7c28afcaa6049609ae6ad74fd3f3554; no voters:
I20260927 07:11:23.270092 17593 raft_consensus.cc:2781] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:23.270530 17593 raft_consensus.cc:686] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 1 LEADER]: Becoming Leader. State: Replica: 588f907cea3c42628a9d6b14d56c53e9, State: Running, Role: LEADER
I20260927 07:11:23.271082 17593 consensus_queue.cc:237] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } }
I20260927 07:11:23.275986 17119 catalog_manager.cc:5495] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 588f907cea3c42628a9d6b14d56c53e9 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "588f907cea3c42628a9d6b14d56c53e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "588f907cea3c42628a9d6b14d56c53e9" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 39735 } health_report { overall_health: HEALTHY } } }
I20260927 07:11:23.313282 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.308051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.30.16.66:39735)
0927 07:11:23.308291 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.308296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.308315 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:11:23.308435 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.308439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.308536 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.308670 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:23.309023 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.309720 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.311111 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.311537 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.311631 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.312120 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:23.312639 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:23.312773 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.312904 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.312966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:11:23.322060 17523 raft_consensus.cc:1261] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 1 FOLLOWER]: Refusing update from remote peer 588f907cea3c42628a9d6b14d56c53e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:23.322077 17262 raft_consensus.cc:1261] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 1 FOLLOWER]: Refusing update from remote peer 588f907cea3c42628a9d6b14d56c53e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:23.323540 17597 consensus_queue.cc:1035] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c28afcaa6049609ae6ad74fd3f3554" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 44849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:23.323971 17593 consensus_queue.cc:1035] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "742f567e91cb40d792a0bc2506fe901c" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 45035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:23.354044 17119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:23.354331 17119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:23.360482 17119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba928e5c0a754db1862b46441358cd2f: Sending DeleteTablet for 3 replicas of tablet 686efcf61a91443cae0d8602dfcce52a
I20260927 07:11:23.361894 17242 tablet_service.cc:1514] Processing DeleteTablet for tablet 686efcf61a91443cae0d8602dfcce52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:23 UTC) from {username='slave'} at 127.0.0.1:35268
I20260927 07:11:23.361909 17373 tablet_service.cc:1514] Processing DeleteTablet for tablet 686efcf61a91443cae0d8602dfcce52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:23 UTC) from {username='slave'} at 127.0.0.1:58188
I20260927 07:11:23.362926 17503 tablet_service.cc:1514] Processing DeleteTablet for tablet 686efcf61a91443cae0d8602dfcce52a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:23 UTC) from {username='slave'} at 127.0.0.1:52842
I20260927 07:11:23.367902 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17180
I20260927 07:11:23.374043 17610 tablet_replica.cc:330] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: stopping tablet replica
I20260927 07:11:23.375798 17610 raft_consensus.cc:2229] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:23.376471 17610 raft_consensus.cc:2258] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:23.377911 17609 tablet_replica.cc:330] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: stopping tablet replica
I20260927 07:11:23.378468 17609 raft_consensus.cc:2229] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:23.378924 17611 tablet_replica.cc:330] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: stopping tablet replica
I20260927 07:11:23.379285 17609 raft_consensus.cc:2258] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:23.379360 17611 raft_consensus.cc:2229] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:23.380631 17611 raft_consensus.cc:2258] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:23.382097 17610 ts_tablet_manager.cc:1905] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:23.382725 17611 ts_tablet_manager.cc:1905] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:23.388464 17609 ts_tablet_manager.cc:1905] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:23.390727 17611 ts_tablet_manager.cc:1918] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:23.390991 17611 log.cc:1196] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/686efcf61a91443cae0d8602dfcce52a
I20260927 07:11:23.391580 17611 ts_tablet_manager.cc:1939] T 686efcf61a91443cae0d8602dfcce52a P 742f567e91cb40d792a0bc2506fe901c: Deleting consensus metadata
I20260927 07:11:23.393786 17108 catalog_manager.cc:4841] TS 742f567e91cb40d792a0bc2506fe901c (127.30.16.67:45035): tablet 686efcf61a91443cae0d8602dfcce52a (table test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]) successfully deleted
I20260927 07:11:23.396296 17609 ts_tablet_manager.cc:1918] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:23.396495 17609 log.cc:1196] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/686efcf61a91443cae0d8602dfcce52a
I20260927 07:11:23.397119 17609 ts_tablet_manager.cc:1939] T 686efcf61a91443cae0d8602dfcce52a P 588f907cea3c42628a9d6b14d56c53e9: Deleting consensus metadata
I20260927 07:11:23.402159 17610 ts_tablet_manager.cc:1918] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:23.402367 17610 log.cc:1196] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/686efcf61a91443cae0d8602dfcce52a
I20260927 07:11:23.403265 17106 catalog_manager.cc:4841] TS 588f907cea3c42628a9d6b14d56c53e9 (127.30.16.66:39735): tablet 686efcf61a91443cae0d8602dfcce52a (table test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]) successfully deleted
I20260927 07:11:23.403731 17610 ts_tablet_manager.cc:1939] T 686efcf61a91443cae0d8602dfcce52a P e7c28afcaa6049609ae6ad74fd3f3554: Deleting consensus metadata
I20260927 07:11:23.405849 17108 catalog_manager.cc:4841] TS e7c28afcaa6049609ae6ad74fd3f3554 (127.30.16.65:44849): tablet 686efcf61a91443cae0d8602dfcce52a (table test-table [id=80ce1e30dbe542f6895e3628e9a5ad9d]) successfully deleted
I20260927 07:11:23.408490 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.369685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.30.16.65:44849)
0927 07:11:23.369927 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.369936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.385516 (+ 15580us) server_negotiation.cc:407] Connection header received
0927 07:11:23.386055 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.386061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.386177 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.386251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:23.386998 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.388122 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.392193 (+ 4071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.392816 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.392932 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.406688 (+ 13756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:23.406701 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:23.406719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:23.407513 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:23.407592 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:23.407627 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:23.407770 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.407992 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.408121 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:11:23.409296 17302 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:23.509481 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17180
I20260927 07:11:23.528870 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17311
I20260927 07:11:23.537024 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.529831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.30.16.66:39735)
0927 07:11:23.530077 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.530085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.530123 (+ 38us) server_negotiation.cc:407] Connection header received
0927 07:11:23.530212 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.530245 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.530369 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.530518 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:23.530957 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.532319 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.533697 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.534171 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.534313 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.535594 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:23.535602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:23.535617 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:23.536247 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:23.536297 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:23.536318 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:23.536433 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.536573 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.536674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 07:11:23.537882 17433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:23.654299 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17311
I20260927 07:11:23.674130 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17441
I20260927 07:11:23.681788 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:23.675016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.30.16.67:45035)
0927 07:11:23.675244 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:23.675251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:23.675273 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:11:23.675451 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:23.675458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:23.675559 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:23.675673 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:23.676081 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.676915 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.678374 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:23.678895 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:23.679034 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:23.680270 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:23.680279 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:23.680296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:23.680953 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:23.681006 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:23.681029 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:23.681161 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:23.681306 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 07:11:23.681433 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 07:11:23.682538 17563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:23.776980 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17441
I20260927 07:11:23.794275 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17090
I20260927 07:11:23.795482 17151 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:23.906041 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17090
2026-09-27T07:11:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3074 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:24 dist-test-slave-8vqf krb5kdc[17619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:24 dist-test-slave-8vqf krb5kdc[17619](info): set up 2 sockets
Sep 27 07:11:24 dist-test-slave-8vqf krb5kdc[17619](info): commencing operation
krb5kdc: starting...
W20260927 07:11:25.043056 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:25 dist-test-slave-8vqf krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:11:25.298861 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37821
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:25.641508 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:25.641884 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:25.642051 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:25.649462 17632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:25.649634 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:25.649752 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:25.649850 17632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:25.649930 17632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:25.660003 17632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:25.661458 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:25.662874 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:25.675271 17637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:25.675235 17640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:25.675304 17638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:25.676622 17632 server_base.cc:1034] running on GCE node
I20260927 07:11:25.677479 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:25.679083 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:25.680298 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790493085680220 us; error 61 us; skew 500 ppm
I20260927 07:11:25.680682 17632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:25 dist-test-slave-8vqf krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:25.684000 17632 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:11:25.685601 17632 webserver.cc:460] Webserver started at http://127.30.16.126:41871/ using document root <none> and password file <none>
I20260927 07:11:25.686281 17632 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:25.686390 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:25.686730 17632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:25.689925 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "e0e1f2353c854c8ca4b4c700788c8daf"
format_stamp: "Formatted at 2026-09-27 07:11:25 on dist-test-slave-8vqf"
server_key: "fa692e69caacb7a1417d8e80fd23b0d0"
server_key_iv: "626787315c8525983bb45bb288b4a09f"
server_key_version: "encryptionkey@0"
I20260927 07:11:25.690725 17632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "e0e1f2353c854c8ca4b4c700788c8daf"
format_stamp: "Formatted at 2026-09-27 07:11:25 on dist-test-slave-8vqf"
server_key: "fa692e69caacb7a1417d8e80fd23b0d0"
server_key_iv: "626787315c8525983bb45bb288b4a09f"
server_key_version: "encryptionkey@0"
I20260927 07:11:25.696030 17632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:11:25.699747 17647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:25.701156 17632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:25.701350 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "e0e1f2353c854c8ca4b4c700788c8daf"
format_stamp: "Formatted at 2026-09-27 07:11:25 on dist-test-slave-8vqf"
server_key: "fa692e69caacb7a1417d8e80fd23b0d0"
server_key_iv: "626787315c8525983bb45bb288b4a09f"
server_key_version: "encryptionkey@0"
I20260927 07:11:25.701555 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:25.725914 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:25.730094 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:25.730399 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:25.749356 17632 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:37821
I20260927 07:11:25.749362 17698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:37821 every 8 connection(s)
I20260927 07:11:25.750816 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:25.754115 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17632
I20260927 07:11:25.754328 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:25.754812 17699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:25.754848 30785 external_mini_cluster.cc:1428] Setting key d0430443e0869d8b6b57a4aad7099afa
I20260927 07:11:25.769635 17699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: Bootstrap starting.
Sep 27 07:11:25 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:25.773542 17699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:25.774761 17699 log.cc:824] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:25.778247 17699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: No bootstrap required, opened a new log
I20260927 07:11:25.778856 17702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:25.756705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36374 (local address 127.30.16.126:37821)
0927 07:11:25.757535 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:25.757549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:25.757599 (+ 50us) server_negotiation.cc:407] Connection header received
0927 07:11:25.759886 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:25.759916 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:25.760501 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:25.760990 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:25.762321 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:25.764353 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:25.765617 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:25.766808 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:25.767013 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:25.771281 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:25.771288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:25.771303 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:25.771349 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:25.774487 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:25.775449 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:25.775464 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:25.775473 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:25.775714 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:25.776074 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:25.776082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:25.776088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:25.776659 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:25.776896 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:25.777341 (+ 445us) server_negotiation.cc:299] Negotiation successful
0927 07:11:25.777688 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":319,"threads_started":1}
I20260927 07:11:25.787216 17699 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } }
I20260927 07:11:25.787679 17699 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:25.787796 17699 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e1f2353c854c8ca4b4c700788c8daf, State: Initialized, Role: FOLLOWER
I20260927 07:11:25.788395 17699 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } }
I20260927 07:11:25.788643 17699 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:25.788779 17699 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:25.788954 17699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:25.791416 17699 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } }
I20260927 07:11:25.792008 17699 leader_election.cc:302] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e1f2353c854c8ca4b4c700788c8daf; no voters:
I20260927 07:11:25.792613 17699 leader_election.cc:288] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:25.792763 17704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:25.793447 17704 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [term 1 LEADER]: Becoming Leader. State: Replica: e0e1f2353c854c8ca4b4c700788c8daf, State: Running, Role: LEADER
I20260927 07:11:25.794068 17704 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } }
I20260927 07:11:25.795055 17699 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:25.797097 17706 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e1f2353c854c8ca4b4c700788c8daf. Latest consensus state: current_term: 1 leader_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } } }
I20260927 07:11:25.797461 17706 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:25.797907 17705 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e1f2353c854c8ca4b4c700788c8daf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37821 } } }
I20260927 07:11:25.798190 17705 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:25.800253 17710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:25.804843 17710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:25.814747 17710 catalog_manager.cc:1347] Generated new cluster ID: 6eda62b833074d86b0da32d1ea26d12c
I20260927 07:11:25.814885 17710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:25.827672 17710 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:25.829042 17710 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:25.837649 17710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: Generated new TSK 0
I20260927 07:11:25.838533 17710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.30.16.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
Entry for principal kudu/127.30.16.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab.
I20260927 07:11:25.902397 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:26.299664 17725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:26.300091 17725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:26.300266 17725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:26.308312 17725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:26.308498 17725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:26.308681 17725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:26.319144 17725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.65.keytab
--principal=kudu/127.30.16.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:26.320581 17725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:26.321985 17725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:26.336136 17730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:26.336113 17733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:26.336050 17731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:26.336845 17725 server_base.cc:1034] running on GCE node
I20260927 07:11:26.337538 17725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:26.338647 17725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:26.339885 17725 hybrid_clock.cc:648] HybridClock initialized: now 1790493086339824 us; error 56 us; skew 500 ppm
I20260927 07:11:26.340238 17725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:26 dist-test-slave-8vqf krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493086, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:26.343575 17725 init.cc:377] Logged in from keytab as kudu/127.30.16.65@KRBTEST.COM (short username kudu)
I20260927 07:11:26.345045 17725 webserver.cc:460] Webserver started at http://127.30.16.65:37119/ using document root <none> and password file <none>
I20260927 07:11:26.345767 17725 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:26.345920 17725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:26.346239 17725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:26.349450 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "7ba8626655834886b2bcbc603393d97e"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "aaa701685e1d54da29520dd5d79d7550"
server_key_iv: "412ae75d6f7cba6c8c39c2e75e98c342"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.350260 17725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "7ba8626655834886b2bcbc603393d97e"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "aaa701685e1d54da29520dd5d79d7550"
server_key_iv: "412ae75d6f7cba6c8c39c2e75e98c342"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.355829 17725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 07:11:26.359440 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.360786 17725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:26.360973 17725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "7ba8626655834886b2bcbc603393d97e"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "aaa701685e1d54da29520dd5d79d7550"
server_key_iv: "412ae75d6f7cba6c8c39c2e75e98c342"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.361183 17725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:26.380473 17725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:26.385167 17725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:26.385480 17725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:26.386507 17725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:26.388031 17725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:26.388106 17725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.388265 17725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:26.388327 17725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.421082 17725 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:36645
I20260927 07:11:26.421129 17852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:36645 every 8 connection(s)
I20260927 07:11:26.422508 17725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:26.432459 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17725
I20260927 07:11:26.432667 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:26.433100 30785 external_mini_cluster.cc:1428] Setting key 808d2b4274377ef0037827fffdb75f7a
Sep 27 07:11:26 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493086, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.65@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:26.445138 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:26.425385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:43549 (local address 127.30.16.126:37821)
0927 07:11:26.425809 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:26.425813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:26.426997 (+ 1184us) server_negotiation.cc:407] Connection header received
0927 07:11:26.428942 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:26.428947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:26.429045 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:26.429167 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:26.430864 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.431559 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.433379 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.433947 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.434093 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:26.438922 (+ 4829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:26.438928 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:26.438931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:26.438968 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:26.441834 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:26.442713 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:26.442718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:26.442720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:26.442866 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:26.443351 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:26.443355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:26.443357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:26.443630 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:26.443792 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:26.444603 (+ 811us) server_negotiation.cc:299] Negotiation successful
0927 07:11:26.444838 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":211,"threads_started":1}
I20260927 07:11:26.446719 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:26.426015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37821 (local address 127.30.16.65:43549)
0927 07:11:26.426816 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:26.426863 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:26.428638 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:26.429435 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:26.429444 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:26.430006 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:26.430671 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:26.430690 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.431734 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.431740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:26.433211 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:26.433229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.434191 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.434209 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:26.434678 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:26.435510 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:26.435549 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:26.438469 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:26.442014 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:26.442025 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:26.442046 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:26.442522 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:26.442996 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:26.443002 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:26.443005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:26.443218 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:26.443763 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:26.443772 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:26.444155 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 07:11:26.444472 (+ 317us) client_negotiation.cc:240] Negotiation successful
0927 07:11:26.445129 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":289,"threads_started":1}
I20260927 07:11:26.448484 17853 heartbeater.cc:347] Connected to a master server at 127.30.16.126:37821
I20260927 07:11:26.448837 17853 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:26.449779 17853 heartbeater.cc:511] Master 127.30.16.126:37821 requested a full tablet report, sending...
I20260927 07:11:26.452203 17664 ts_manager.cc:194] Registered new tserver with Master: 7ba8626655834886b2bcbc603393d97e (127.30.16.65:36645)
I20260927 07:11:26.453817 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.65@KRBTEST.COM'} at 127.30.16.65:43549
WARNING: no policy specified for kudu/127.30.16.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
Entry for principal kudu/127.30.16.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab.
I20260927 07:11:26.494889 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:26.829067 17859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:26.829366 17859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:26.829548 17859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:26.836838 17859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:26.836961 17859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:26.837159 17859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:26.847357 17859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.66.keytab
--principal=kudu/127.30.16.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:26.848739 17859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:26.850109 17859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:26.863550 17865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:26.863590 17867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:26.863524 17864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:26.864535 17859 server_base.cc:1034] running on GCE node
I20260927 07:11:26.865180 17859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:26.866268 17859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:26.867494 17859 hybrid_clock.cc:648] HybridClock initialized: now 1790493086867406 us; error 66 us; skew 500 ppm
I20260927 07:11:26.867832 17859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:26 dist-test-slave-8vqf krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493086, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:26.870985 17859 init.cc:377] Logged in from keytab as kudu/127.30.16.66@KRBTEST.COM (short username kudu)
I20260927 07:11:26.872611 17859 webserver.cc:460] Webserver started at http://127.30.16.66:39111/ using document root <none> and password file <none>
I20260927 07:11:26.873342 17859 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:26.873473 17859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:26.873802 17859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:26.876897 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "119a51b4cd444c0eb57efaa654ae46a3"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "5f9157a6aabac3d668d18b98755230ec"
server_key_iv: "514b6d16c190f5adc0a8f242b2806159"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.877691 17859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "119a51b4cd444c0eb57efaa654ae46a3"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "5f9157a6aabac3d668d18b98755230ec"
server_key_iv: "514b6d16c190f5adc0a8f242b2806159"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.883227 17859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:11:26.886943 17874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.888427 17859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:26.888619 17859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "119a51b4cd444c0eb57efaa654ae46a3"
format_stamp: "Formatted at 2026-09-27 07:11:26 on dist-test-slave-8vqf"
server_key: "5f9157a6aabac3d668d18b98755230ec"
server_key_iv: "514b6d16c190f5adc0a8f242b2806159"
server_key_version: "encryptionkey@0"
I20260927 07:11:26.888847 17859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:26.913940 17859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:26.918162 17859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:26.918481 17859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:26.919534 17859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:26.920938 17859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:26.921046 17859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.921203 17859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:26.921277 17859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:26.955909 17859 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:37425
I20260927 07:11:26.955998 17986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:37425 every 8 connection(s)
I20260927 07:11:26.957330 17859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:26.960320 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17859
I20260927 07:11:26.960527 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:26.960988 30785 external_mini_cluster.cc:1428] Setting key 75bb7d8c8090e9fc42fba1b25f781ac6
Sep 27 07:11:26 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493086, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.66@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:26.982275 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:26.960047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:39251 (local address 127.30.16.126:37821)
0927 07:11:26.960576 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:26.960583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:26.961786 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 07:11:26.964246 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:26.964251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:26.964316 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:26.964415 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:26.966682 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.967425 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.969977 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.970398 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.970490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:26.974845 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:26.974854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:26.974862 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:26.974912 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:26.978648 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:26.979522 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:26.979531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:26.979537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:26.979709 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:26.980160 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:26.980168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:26.980174 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:26.980583 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:26.980739 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:26.981493 (+ 754us) server_negotiation.cc:299] Negotiation successful
0927 07:11:26.981762 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":240,"threads_started":1}
I20260927 07:11:26.983035 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:26.960769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37821 (local address 127.30.16.66:39251)
0927 07:11:26.961610 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:26.961666 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:26.963928 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:26.964711 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:26.964725 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:26.965580 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:26.966455 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:26.966484 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.967585 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.967593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:26.969821 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:26.969841 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:26.970670 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:26.970680 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:26.971024 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:26.971789 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:26.971832 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:26.974531 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:26.978809 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:26.978817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:26.978832 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:26.979372 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:26.979833 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:26.979837 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:26.979839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:26.980033 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:26.980740 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:26.980747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:26.981090 (+ 343us) client_negotiation.cc:769] Sending connection context
0927 07:11:26.981352 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 07:11:26.981778 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":317,"threads_started":1}
I20260927 07:11:26.984532 17987 heartbeater.cc:347] Connected to a master server at 127.30.16.126:37821
I20260927 07:11:26.984884 17987 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:26.985805 17987 heartbeater.cc:511] Master 127.30.16.126:37821 requested a full tablet report, sending...
I20260927 07:11:26.987527 17664 ts_manager.cc:194] Registered new tserver with Master: 119a51b4cd444c0eb57efaa654ae46a3 (127.30.16.66:37425)
I20260927 07:11:26.988384 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.66@KRBTEST.COM'} at 127.30.16.66:39251
WARNING: no policy specified for kudu/127.30.16.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
Entry for principal kudu/127.30.16.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab.
I20260927 07:11:27.024075 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:27.365996 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:27.366353 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:27.366528 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:27.375618 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:27.375797 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:27.375983 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:27.386183 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.67.keytab
--principal=kudu/127.30.16.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:27.387801 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:27.389230 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:27.404071 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:27.404281 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:27.404469 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:27.405481 17993 server_base.cc:1034] running on GCE node
I20260927 07:11:27.406256 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:27.407419 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:27.408670 17993 hybrid_clock.cc:648] HybridClock initialized: now 1790493087408612 us; error 53 us; skew 500 ppm
I20260927 07:11:27.409051 17993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:27 dist-test-slave-8vqf krb5kdc[17619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493087, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:27.412492 17993 init.cc:377] Logged in from keytab as kudu/127.30.16.67@KRBTEST.COM (short username kudu)
I20260927 07:11:27.414113 17993 webserver.cc:460] Webserver started at http://127.30.16.67:33993/ using document root <none> and password file <none>
I20260927 07:11:27.414915 17993 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:27.415069 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:27.415391 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:27.418406 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "219a01f15f1b4730bd92a2d9de8646d0"
format_stamp: "Formatted at 2026-09-27 07:11:27 on dist-test-slave-8vqf"
server_key: "aaf0f1e2a6487876802e70d1cdfb3098"
server_key_iv: "25b8fee49fb890dfefddf366d851c031"
server_key_version: "encryptionkey@0"
I20260927 07:11:27.419255 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "219a01f15f1b4730bd92a2d9de8646d0"
format_stamp: "Formatted at 2026-09-27 07:11:27 on dist-test-slave-8vqf"
server_key: "aaf0f1e2a6487876802e70d1cdfb3098"
server_key_iv: "25b8fee49fb890dfefddf366d851c031"
server_key_version: "encryptionkey@0"
I20260927 07:11:27.424784 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:27.428380 18008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:27.429950 17993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:27.430163 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "219a01f15f1b4730bd92a2d9de8646d0"
format_stamp: "Formatted at 2026-09-27 07:11:27 on dist-test-slave-8vqf"
server_key: "aaf0f1e2a6487876802e70d1cdfb3098"
server_key_iv: "25b8fee49fb890dfefddf366d851c031"
server_key_version: "encryptionkey@0"
I20260927 07:11:27.430397 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:27.448051 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:27.452595 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:27.452908 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:27.453933 17993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:27.455435 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:27.455538 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:27.455686 17993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:27.455768 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:27.457823 17853 heartbeater.cc:503] Master 127.30.16.126:37821 was elected leader, sending a full tablet report...
I20260927 07:11:27.490676 17993 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:39319
I20260927 07:11:27.490726 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:39319 every 8 connection(s)
I20260927 07:11:27.492039 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:27.499882 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 17993
I20260927 07:11:27.500072 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:27.500509 30785 external_mini_cluster.cc:1428] Setting key 80dadbc88c62525caa045afbe7d11ab2
Sep 27 07:11:27 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493087, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.67@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:27.514767 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.495165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:48265 (local address 127.30.16.126:37821)
0927 07:11:27.495974 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.495979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.496995 (+ 1016us) server_negotiation.cc:407] Connection header received
0927 07:11:27.498968 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.498973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.499080 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.499210 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:27.501016 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.501950 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.504092 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.504607 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.504786 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.508516 (+ 3730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:27.508520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:27.508524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:27.508559 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:27.511636 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:27.512416 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:27.512421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:27.512424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:27.512551 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:27.512958 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:27.512961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:27.512963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:27.513240 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:27.513366 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.514078 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.514302 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":564,"threads_started":1}
I20260927 07:11:27.515908 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.496061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37821 (local address 127.30.16.67:48265)
0927 07:11:27.496800 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.496862 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.498671 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.499473 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.499482 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.500098 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:27.500802 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.500830 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.502129 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.502134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.503959 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.503972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.504872 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.504877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.505126 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.505877 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:11:27.505901 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:27.508295 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:27.511794 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:27.511801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:27.511823 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:27.512259 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:27.512649 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:11:27.512652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:11:27.512654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:11:27.512854 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:11:27.513345 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:27.513352 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:27.513682 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.513919 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.514430 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":299,"threads_started":1}
I20260927 07:11:27.517446 18121 heartbeater.cc:347] Connected to a master server at 127.30.16.126:37821
I20260927 07:11:27.517756 18121 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:27.518518 18121 heartbeater.cc:511] Master 127.30.16.126:37821 requested a full tablet report, sending...
I20260927 07:11:27.519981 17664 ts_manager.cc:194] Registered new tserver with Master: 219a01f15f1b4730bd92a2d9de8646d0 (127.30.16.67:39319)
I20260927 07:11:27.520808 17664 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.30.16.67@KRBTEST.COM'} at 127.30.16.67:48265
I20260927 07:11:27.528868 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:27.546249 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.533198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.30.16.126:37821)
0927 07:11:27.533380 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.533385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.533766 (+ 381us) server_negotiation.cc:407] Connection header received
0927 07:11:27.533905 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.533908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.533980 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.534284 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:27.535716 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.536554 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.537853 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.538383 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.538515 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.541034 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:27.541039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:27.541042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:27.541080 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:27.543869 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:27.544497 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:27.544501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:27.544503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:27.544626 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:27.544961 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:27.544964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:27.544966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:27.545291 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:27.545486 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.545807 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.545995 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:11:27.550489 17664 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:27.557473 17664 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:27.582966 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.570752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.30.16.65:36645)
0927 07:11:27.572766 (+ 2014us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.572778 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.573200 (+ 422us) server_negotiation.cc:407] Connection header received
0927 07:11:27.573318 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.573326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.573877 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.574096 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:27.575682 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.577213 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.579992 (+ 2779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.581365 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.581509 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.581758 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.582357 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.582538 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805,"thread_start_us":1642,"threads_started":1}
I20260927 07:11:27.582998 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.573843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:39319 (local address 127.0.0.1:51816)
0927 07:11:27.575054 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.575077 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.575219 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.576034 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.576039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.576088 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:27.576425 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.576435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.578502 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.578508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.580461 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.580471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.582136 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.582142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.582293 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.582320 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.582416 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.582539 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":23,"thread_start_us":873,"threads_started":1}
I20260927 07:11:27.583477 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.570150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.30.16.66:37425)
0927 07:11:27.573272 (+ 3122us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.573279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.573301 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:11:27.573429 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.573434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.573861 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.574070 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:27.575381 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.577040 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.579810 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.581360 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.581509 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.581727 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.582901 (+ 1174us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.583107 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2971,"thread_start_us":2838,"threads_started":1}
I20260927 07:11:27.583899 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.573264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51816 (local address 127.30.16.67:39319)
0927 07:11:27.575538 (+ 2274us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.575547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.575581 (+ 34us) server_negotiation.cc:407] Connection header received
0927 07:11:27.575688 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.575694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.575860 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.576054 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:27.576601 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.578370 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.580605 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.581591 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.581725 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.581985 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.583461 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.583642 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2093,"thread_start_us":1795,"threads_started":1}
I20260927 07:11:27.583933 18136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.570200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36645 (local address 127.0.0.1:47216)
0927 07:11:27.572921 (+ 2721us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.572940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.573065 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.574400 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.574408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.574466 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:27.575496 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.575510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.577355 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.577361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.579834 (+ 2473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.579849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.581952 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.581958 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.582147 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.582185 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.583423 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.583525 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2513,"mutex_wait_us":56,"spinlock_wait_cycles":2816,"thread_start_us":1270,"threads_started":1}
I20260927 07:11:27.582777 18135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.570101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:37425 (local address 127.0.0.1:60760)
0927 07:11:27.571981 (+ 1880us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.572037 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.572248 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.574394 (+ 2146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.574401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.574430 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:27.574988 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.575010 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.577179 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.577185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.579640 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.579652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.581587 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.581593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.581806 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.581858 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.582012 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.582238 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":65,"thread_start_us":2476,"threads_started":1}
I20260927 07:11:27.587600 17788 tablet_service.cc:1467] Processing CreateTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 (DEFAULT_TABLE table=test-table [id=63353d1879774c2a93b52183e6950362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:27.587628 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 (DEFAULT_TABLE table=test-table [id=63353d1879774c2a93b52183e6950362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:27.587632 17922 tablet_service.cc:1467] Processing CreateTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 (DEFAULT_TABLE table=test-table [id=63353d1879774c2a93b52183e6950362]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:27.589134 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e382b0e9e04c66ad633e9e704b19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:27.589139 17922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e382b0e9e04c66ad633e9e704b19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:27.589442 17788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e382b0e9e04c66ad633e9e704b19e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:27.603595 18140 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Bootstrap starting.
I20260927 07:11:27.604269 18141 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Bootstrap starting.
I20260927 07:11:27.605510 18142 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Bootstrap starting.
I20260927 07:11:27.608026 18141 tablet_bootstrap.cc:654] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:27.608024 18140 tablet_bootstrap.cc:654] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:27.608624 18142 tablet_bootstrap.cc:654] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:27.609380 18140 log.cc:824] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:27.609396 18141 log.cc:824] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:27.609678 18142 log.cc:824] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:27.612874 18141 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: No bootstrap required, opened a new log
I20260927 07:11:27.612967 18142 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: No bootstrap required, opened a new log
I20260927 07:11:27.613183 18141 ts_tablet_manager.cc:1397] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 07:11:27.613221 18140 tablet_bootstrap.cc:492] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: No bootstrap required, opened a new log
I20260927 07:11:27.613236 18142 ts_tablet_manager.cc:1397] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 07:11:27.613461 18140 ts_tablet_manager.cc:1397] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 07:11:27.620658 18141 raft_consensus.cc:348] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.621129 18141 raft_consensus.cc:374] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:27.621254 18141 raft_consensus.cc:729] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219a01f15f1b4730bd92a2d9de8646d0, State: Initialized, Role: FOLLOWER
I20260927 07:11:27.622013 18141 consensus_queue.cc:260] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.623473 18121 heartbeater.cc:503] Master 127.30.16.126:37821 was elected leader, sending a full tablet report...
I20260927 07:11:27.624238 18141 ts_tablet_manager.cc:1428] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260927 07:11:27.624420 18142 raft_consensus.cc:348] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.624866 18142 raft_consensus.cc:374] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:27.625018 18142 raft_consensus.cc:729] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a51b4cd444c0eb57efaa654ae46a3, State: Initialized, Role: FOLLOWER
I20260927 07:11:27.625241 18140 raft_consensus.cc:348] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.625825 18140 raft_consensus.cc:374] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:27.625686 18142 consensus_queue.cc:260] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.625936 18140 raft_consensus.cc:729] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba8626655834886b2bcbc603393d97e, State: Initialized, Role: FOLLOWER
I20260927 07:11:27.626717 18140 consensus_queue.cc:260] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.626948 17987 heartbeater.cc:503] Master 127.30.16.126:37821 was elected leader, sending a full tablet report...
I20260927 07:11:27.628510 18140 ts_tablet_manager.cc:1428] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260927 07:11:27.628921 18142 ts_tablet_manager.cc:1428] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260927 07:11:27.675032 17854 tablet.cc:2369] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:27.709460 17988 tablet.cc:2369] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 07:11:27.744021 18122 tablet.cc:2369] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:27.843617 18147 raft_consensus.cc:482] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:27.843914 18147 raft_consensus.cc:504] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.845607 18147 leader_election.cc:288] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ba8626655834886b2bcbc603393d97e (127.30.16.65:36645), 219a01f15f1b4730bd92a2d9de8646d0 (127.30.16.67:39319)
I20260927 07:11:27.852563 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.846264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:39077 (local address 127.30.16.67:39319)
0927 07:11:27.846457 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.846465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.847069 (+ 604us) server_negotiation.cc:407] Connection header received
0927 07:11:27.847272 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.847281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.847342 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.847438 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:27.848196 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.848928 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.850762 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.851513 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.851707 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.851790 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.852158 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.852289 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:11:27.852573 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.846105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:58305 (local address 127.30.16.65:36645)
0927 07:11:27.846406 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.846414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.847032 (+ 618us) server_negotiation.cc:407] Connection header received
0927 07:11:27.847328 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.847332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.847395 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.847496 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:27.848126 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.848853 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.850768 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.851561 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.851750 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.851831 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.852172 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.852301 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 07:11:27.852813 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.846045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:39319 (local address 127.30.16.66:39077)
0927 07:11:27.846961 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.846975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.847135 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.847465 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.847469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.847494 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:27.848043 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.848051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.849060 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.849067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.850603 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.850613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.851749 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.851755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.851991 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.852028 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.852132 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.852273 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"mutex_wait_us":113,"thread_start_us":551,"threads_started":1}
I20260927 07:11:27.852787 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.845989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:36645 (local address 127.30.16.66:58305)
0927 07:11:27.846912 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:27.846928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:27.847174 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:27.847508 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:27.847513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:27.847535 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:27.847979 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.847988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.848985 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.848992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.850614 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:27.850634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.851795 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.851801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:27.851988 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.852026 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 07:11:27.852146 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 07:11:27.852277 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":42,"thread_start_us":435,"threads_started":1}
I20260927 07:11:27.853960 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e382b0e9e04c66ad633e9e704b19e3" candidate_uuid: "119a51b4cd444c0eb57efaa654ae46a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba8626655834886b2bcbc603393d97e" is_pre_election: true
I20260927 07:11:27.854015 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e382b0e9e04c66ad633e9e704b19e3" candidate_uuid: "119a51b4cd444c0eb57efaa654ae46a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a01f15f1b4730bd92a2d9de8646d0" is_pre_election: true
I20260927 07:11:27.854478 18076 raft_consensus.cc:2454] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a51b4cd444c0eb57efaa654ae46a3 in term 0.
I20260927 07:11:27.854480 17808 raft_consensus.cc:2454] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a51b4cd444c0eb57efaa654ae46a3 in term 0.
I20260927 07:11:27.855201 17878 leader_election.cc:302] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a51b4cd444c0eb57efaa654ae46a3, 219a01f15f1b4730bd92a2d9de8646d0; no voters:
I20260927 07:11:27.855598 18147 raft_consensus.cc:2781] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:27.855746 18147 raft_consensus.cc:482] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:27.855855 18147 raft_consensus.cc:3037] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:27.858039 18147 raft_consensus.cc:504] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.858651 18147 leader_election.cc:288] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [CANDIDATE]: Term 1 election: Requested vote from peers 7ba8626655834886b2bcbc603393d97e (127.30.16.65:36645), 219a01f15f1b4730bd92a2d9de8646d0 (127.30.16.67:39319)
I20260927 07:11:27.859179 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e382b0e9e04c66ad633e9e704b19e3" candidate_uuid: "119a51b4cd444c0eb57efaa654ae46a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219a01f15f1b4730bd92a2d9de8646d0"
I20260927 07:11:27.859172 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e382b0e9e04c66ad633e9e704b19e3" candidate_uuid: "119a51b4cd444c0eb57efaa654ae46a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ba8626655834886b2bcbc603393d97e"
I20260927 07:11:27.859501 18076 raft_consensus.cc:3037] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:27.859499 17808 raft_consensus.cc:3037] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:27.861701 17808 raft_consensus.cc:2454] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a51b4cd444c0eb57efaa654ae46a3 in term 1.
I20260927 07:11:27.861712 18076 raft_consensus.cc:2454] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a51b4cd444c0eb57efaa654ae46a3 in term 1.
I20260927 07:11:27.862143 17875 leader_election.cc:302] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a51b4cd444c0eb57efaa654ae46a3, 7ba8626655834886b2bcbc603393d97e; no voters:
I20260927 07:11:27.862474 18147 raft_consensus.cc:2781] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:27.862983 18147 raft_consensus.cc:686] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 1 LEADER]: Becoming Leader. State: Replica: 119a51b4cd444c0eb57efaa654ae46a3, State: Running, Role: LEADER
I20260927 07:11:27.863454 18147 consensus_queue.cc:237] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } }
I20260927 07:11:27.868368 17664 catalog_manager.cc:5495] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 119a51b4cd444c0eb57efaa654ae46a3 (127.30.16.66). New cstate: current_term: 1 leader_uuid: "119a51b4cd444c0eb57efaa654ae46a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a51b4cd444c0eb57efaa654ae46a3" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 37425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 } health_report { overall_health: UNKNOWN } } }
I20260927 07:11:27.950157 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:27.944684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.30.16.66:37425)
0927 07:11:27.944913 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:27.944920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:27.944939 (+ 19us) server_negotiation.cc:407] Connection header received
0927 07:11:27.945064 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:27.945068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:27.945146 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:27.945304 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:27.945690 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.946406 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.947872 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:27.948399 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:27.948534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:27.949053 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:27.949569 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:27.949735 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:27.949786 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 07:11:27.949887 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:11:27.959583 18076 raft_consensus.cc:1261] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 1 FOLLOWER]: Refusing update from remote peer 119a51b4cd444c0eb57efaa654ae46a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:27.959589 17808 raft_consensus.cc:1261] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 1 FOLLOWER]: Refusing update from remote peer 119a51b4cd444c0eb57efaa654ae46a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:27.960659 18151 consensus_queue.cc:1035] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "219a01f15f1b4730bd92a2d9de8646d0" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 39319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:27.961114 18147 consensus_queue.cc:1035] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ba8626655834886b2bcbc603393d97e" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 36645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:27.978008 18154 mvcc.cc:205] Tried to move back new op lower bound from 7333859688269918208 to 7333859687892353024. Current Snapshot: MvccSnapshot[applied={T|T < 7333859688269918208}]
I20260927 07:11:27.995234 17664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:27.995558 17664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:27.999788 17664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0e1f2353c854c8ca4b4c700788c8daf: Sending DeleteTablet for 3 replicas of tablet 18e382b0e9e04c66ad633e9e704b19e3
I20260927 07:11:28.001345 18056 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:51816
I20260927 07:11:28.001348 17788 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:47216
I20260927 07:11:28.001533 17922 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e382b0e9e04c66ad633e9e704b19e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:27 UTC) from {username='kudu', principal='kudu/127.30.16.126@KRBTEST.COM'} at 127.0.0.1:60760
I20260927 07:11:28.003615 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17725
I20260927 07:11:28.014048 18164 tablet_replica.cc:330] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: stopping tablet replica
I20260927 07:11:28.014809 18166 tablet_replica.cc:330] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: stopping tablet replica
I20260927 07:11:28.015146 18164 raft_consensus.cc:2229] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:28.015409 18166 raft_consensus.cc:2229] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:28.015781 18164 raft_consensus.cc:2258] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:28.016217 18166 raft_consensus.cc:2258] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:28.017683 18165 tablet_replica.cc:330] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: stopping tablet replica
I20260927 07:11:28.018173 18165 raft_consensus.cc:2229] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:28.018795 18165 raft_consensus.cc:2258] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:28.020018 18166 ts_tablet_manager.cc:1905] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:28.020012 18164 ts_tablet_manager.cc:1905] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:28.027413 18165 ts_tablet_manager.cc:1905] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:28.027503 18164 ts_tablet_manager.cc:1918] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:28.027503 18166 ts_tablet_manager.cc:1918] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:28.027729 18164 log.cc:1196] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/18e382b0e9e04c66ad633e9e704b19e3
I20260927 07:11:28.027871 18166 log.cc:1196] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/18e382b0e9e04c66ad633e9e704b19e3
I20260927 07:11:28.028316 18164 ts_tablet_manager.cc:1939] T 18e382b0e9e04c66ad633e9e704b19e3 P 219a01f15f1b4730bd92a2d9de8646d0: Deleting consensus metadata
Sep 27 07:11:28 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.65@KRBTEST.COM
I20260927 07:11:28.030917 18166 ts_tablet_manager.cc:1939] T 18e382b0e9e04c66ad633e9e704b19e3 P 119a51b4cd444c0eb57efaa654ae46a3: Deleting consensus metadata
I20260927 07:11:28.031391 17651 catalog_manager.cc:4841] TS 219a01f15f1b4730bd92a2d9de8646d0 (127.30.16.67:39319): tablet 18e382b0e9e04c66ad633e9e704b19e3 (table test-table [id=63353d1879774c2a93b52183e6950362]) successfully deleted
I20260927 07:11:28.032953 17651 catalog_manager.cc:4841] TS 119a51b4cd444c0eb57efaa654ae46a3 (127.30.16.66:37425): tablet 18e382b0e9e04c66ad633e9e704b19e3 (table test-table [id=63353d1879774c2a93b52183e6950362]) successfully deleted
I20260927 07:11:28.036382 18165 ts_tablet_manager.cc:1918] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:28.036577 18165 log.cc:1196] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/18e382b0e9e04c66ad633e9e704b19e3
I20260927 07:11:28.037232 18165 ts_tablet_manager.cc:1939] T 18e382b0e9e04c66ad633e9e704b19e3 P 7ba8626655834886b2bcbc603393d97e: Deleting consensus metadata
I20260927 07:11:28.038020 18137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:28.005152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47220 (local address 127.30.16.65:36645)
0927 07:11:28.005348 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:28.005355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:28.016939 (+ 11584us) server_negotiation.cc:407] Connection header received
0927 07:11:28.017206 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:28.017213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:28.017333 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:28.017476 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:28.019634 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.020787 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.022897 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.023579 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.023702 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:28.030156 (+ 6454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:28.030171 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:28.030190 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:28.030263 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:28.034793 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.035664 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.035672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.035674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.035823 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.036240 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.036244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.036246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.036766 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:28.037101 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:28.037404 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 07:11:28.037634 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:11:28.038893 17848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:28.039472 17648 catalog_manager.cc:4841] TS 7ba8626655834886b2bcbc603393d97e (127.30.16.65:36645): tablet 18e382b0e9e04c66ad633e9e704b19e3 (table test-table [id=63353d1879774c2a93b52183e6950362]) successfully deleted
I20260927 07:11:28.131273 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17725
I20260927 07:11:28.155535 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17859
Sep 27 07:11:28 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.66@KRBTEST.COM
I20260927 07:11:28.172655 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:28.156365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.30.16.66:37425)
0927 07:11:28.156539 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:28.156547 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:28.156623 (+ 76us) server_negotiation.cc:407] Connection header received
0927 07:11:28.156844 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:28.156850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:28.156946 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:28.157066 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:28.158348 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.159328 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.160620 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.161086 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.161234 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:28.165747 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:28.165762 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:28.165778 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:28.165829 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:28.169336 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.170244 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.170252 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.170254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.170424 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.170918 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.170932 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.170934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.171335 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:28.171716 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:28.172003 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 07:11:28.172256 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:11:28.173452 17982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:28.288578 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17859
I20260927 07:11:28.308421 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17993
Sep 27 07:11:28 dist-test-slave-8vqf krb5kdc[17619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.67@KRBTEST.COM
I20260927 07:11:28.324607 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:28.309154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.30.16.67:39319)
0927 07:11:28.309332 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:28.309339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:28.309401 (+ 62us) server_negotiation.cc:407] Connection header received
0927 07:11:28.309648 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:28.309652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:28.309718 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:28.309835 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:28.311356 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.312093 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.313402 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:28.313803 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:28.313954 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:28.318825 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:28.318842 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:28.318876 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:28.318919 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:28.321562 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.322350 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.322359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.322361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.322507 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:28.322974 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:28.322979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:28.322982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:28.323370 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:28.323727 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:28.324020 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 07:11:28.324252 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 07:11:28.325299 18116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:28.417496 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17993
I20260927 07:11:28.434206 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 17632
I20260927 07:11:28.435317 17694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:28.550683 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 17632
2026-09-27T07:11:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4645 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T07:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:28Z Disabled control of system clock
I20260927 07:11:28.629753 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40497
--webserver_interface=127.30.16.126
--webserver_port=0
--builtin_ntp_servers=127.30.16.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:40497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:28.966359 18175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:28.966696 18175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:28.966779 18175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:28.974210 18175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:28.974327 18175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:28.974371 18175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:28.974421 18175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:28.974457 18175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:28.983857 18175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:40497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:28.985152 18175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:28.987071 18175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:28.999187 18183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:28.999301 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:28.999542 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:29.000293 18175 server_base.cc:1034] running on GCE node
I20260927 07:11:29.001052 18175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:29.002485 18175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:29.003736 18175 hybrid_clock.cc:648] HybridClock initialized: now 1790493089003692 us; error 45 us; skew 500 ppm
I20260927 07:11:29.004117 18175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:29.013940 18175 webserver.cc:460] Webserver started at http://127.30.16.126:40325/ using document root <none> and password file <none>
I20260927 07:11:29.014868 18175 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:29.014971 18175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:29.015252 18175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:29.018266 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "8645ae865b474159951abad8555b0f06"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "3cd544d5d7274f7ee8459da6c8be08b7"
server_key_iv: "e813b825884a7443bdbe4c3399269bea"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.019121 18175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "8645ae865b474159951abad8555b0f06"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "3cd544d5d7274f7ee8459da6c8be08b7"
server_key_iv: "e813b825884a7443bdbe4c3399269bea"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.024567 18175 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:29.028208 18189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:29.029749 18175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:29.029982 18175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "8645ae865b474159951abad8555b0f06"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "3cd544d5d7274f7ee8459da6c8be08b7"
server_key_iv: "e813b825884a7443bdbe4c3399269bea"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.030223 18175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:29.043609 18175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:29.044520 18175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:29.044881 18175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:29.064028 18175 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:40497
I20260927 07:11:29.064047 18240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:40497 every 8 connection(s)
I20260927 07:11:29.065505 18175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:29.069556 18241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:29.075692 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18175
I20260927 07:11:29.075901 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:29.076349 30785 external_mini_cluster.cc:1428] Setting key 16ff6efffd0d6554c26fb78ce294229d
I20260927 07:11:29.081956 18241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: Bootstrap starting.
I20260927 07:11:29.086172 18241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:29.087330 18241 log.cc:824] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:29.090905 18241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: No bootstrap required, opened a new log
I20260927 07:11:29.092842 18244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:29.078410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60814 (local address 127.30.16.126:40497)
0927 07:11:29.079591 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:29.079607 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:29.079660 (+ 53us) server_negotiation.cc:407] Connection header received
0927 07:11:29.082044 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:29.082076 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:29.082751 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:29.083227 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:29.083846 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.085847 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.087250 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.088280 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.088435 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:29.089817 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:29.089831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:29.089855 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:29.090629 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:29.090733 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:29.090765 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:29.090891 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:29.091103 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 07:11:29.091505 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":362,"threads_started":1}
I20260927 07:11:29.099985 18241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } }
I20260927 07:11:29.100353 18241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:29.100473 18241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8645ae865b474159951abad8555b0f06, State: Initialized, Role: FOLLOWER
I20260927 07:11:29.101063 18241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } }
I20260927 07:11:29.101305 18241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:29.101405 18241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:29.101562 18241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:29.104233 18241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } }
I20260927 07:11:29.104707 18241 leader_election.cc:302] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8645ae865b474159951abad8555b0f06; no voters:
I20260927 07:11:29.105240 18241 leader_election.cc:288] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:29.105544 18246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:29.106243 18246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [term 1 LEADER]: Becoming Leader. State: Replica: 8645ae865b474159951abad8555b0f06, State: Running, Role: LEADER
I20260927 07:11:29.106810 18246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } }
I20260927 07:11:29.107559 18241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:29.109578 18247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8645ae865b474159951abad8555b0f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } } }
I20260927 07:11:29.109650 18248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8645ae865b474159951abad8555b0f06. Latest consensus state: current_term: 1 leader_uuid: "8645ae865b474159951abad8555b0f06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8645ae865b474159951abad8555b0f06" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40497 } } }
I20260927 07:11:29.110013 18248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:29.110018 18247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:29.111064 18254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:29.117879 18254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:29.128598 18254 catalog_manager.cc:1347] Generated new cluster ID: 1d301ae2ece145aab1f00956c9984016
I20260927 07:11:29.128732 18254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:29.154731 18254 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:29.156348 18254 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:29.162282 18254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: Generated new TSK 0
I20260927 07:11:29.163262 18254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:29.170289 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.65:0
--local_ip_for_outbound_sockets=127.30.16.65
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--builtin_ntp_servers=127.30.16.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 07:11:29.521046 18265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:29.521345 18265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:29.521536 18265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:29.529426 18265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:29.529541 18265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:29.529709 18265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.65
I20260927 07:11:29.539809 18265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.16.65
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.65
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:29.541282 18265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:29.543298 18265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:29.556888 18271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:29.556876 18270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:29.557077 18273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:29.557744 18265 server_base.cc:1034] running on GCE node
I20260927 07:11:29.558382 18265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:29.559428 18265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:29.560674 18265 hybrid_clock.cc:648] HybridClock initialized: now 1790493089560605 us; error 61 us; skew 500 ppm
I20260927 07:11:29.561100 18265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:29.563526 18265 webserver.cc:460] Webserver started at http://127.30.16.65:38695/ using document root <none> and password file <none>
I20260927 07:11:29.564265 18265 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:29.564404 18265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:29.564742 18265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:29.567847 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data/instance:
uuid: "1aec0d3c40b045acbc52248ebc553005"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "28a6778f8cc95221c260da1d7b06e931"
server_key_iv: "9a0fb13fb62f9b6649c7f6c15533ce36"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.568681 18265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance:
uuid: "1aec0d3c40b045acbc52248ebc553005"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "28a6778f8cc95221c260da1d7b06e931"
server_key_iv: "9a0fb13fb62f9b6649c7f6c15533ce36"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.574144 18265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260927 07:11:29.577749 18279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:29.579218 18265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:29.579391 18265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal
uuid: "1aec0d3c40b045acbc52248ebc553005"
format_stamp: "Formatted at 2026-09-27 07:11:29 on dist-test-slave-8vqf"
server_key: "28a6778f8cc95221c260da1d7b06e931"
server_key_iv: "9a0fb13fb62f9b6649c7f6c15533ce36"
server_key_version: "encryptionkey@0"
I20260927 07:11:29.579628 18265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:29.592067 18265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:29.592970 18265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:29.593291 18265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:29.594331 18265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:29.595903 18265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:29.596025 18265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:29.596161 18265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:29.596275 18265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:29.630033 18265 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.65:33547
I20260927 07:11:29.630082 18391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.65:33547 every 8 connection(s)
I20260927 07:11:29.631491 18265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/data/info.pb
I20260927 07:11:29.636206 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18265
I20260927 07:11:29.636399 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal/instance
I20260927 07:11:29.636934 30785 external_mini_cluster.cc:1428] Setting key 028c5da5a6e3780be84af037512cc31b
I20260927 07:11:29.640189 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.66:0
--local_ip_for_outbound_sockets=127.30.16.66
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--builtin_ntp_servers=127.30.16.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:29.646826 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:29.634291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.65:55067 (local address 127.30.16.126:40497)
0927 07:11:29.634807 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:29.634812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:29.636120 (+ 1308us) server_negotiation.cc:407] Connection header received
0927 07:11:29.638468 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:29.638475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:29.638593 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:29.638723 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:29.640138 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.640980 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.643609 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.644067 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.644174 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:29.645864 (+ 1690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:29.645869 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:29.645873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:29.645893 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:29.645936 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:29.645949 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:29.646020 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:29.646354 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 07:11:29.646450 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":290,"threads_started":1}
I20260927 07:11:29.648231 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:29.634977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:40497 (local address 127.30.16.65:55067)
0927 07:11:29.635934 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:29.635990 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:29.638183 (+ 2193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:29.638984 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:29.638992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:29.639122 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:29.639936 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:29.639963 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.641143 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.641151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:29.643394 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:29.643412 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:29.644304 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:29.644313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:29.644663 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:29.645471 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:29.645477 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:29.645540 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:29.645652 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:29.646052 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:29.646067 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:29.646079 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 07:11:29.646290 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 07:11:29.646777 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":288,"threads_started":1}
I20260927 07:11:29.650141 18392 heartbeater.cc:347] Connected to a master server at 127.30.16.126:40497
I20260927 07:11:29.650473 18392 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:29.651281 18392 heartbeater.cc:511] Master 127.30.16.126:40497 requested a full tablet report, sending...
I20260927 07:11:29.653461 18206 ts_manager.cc:194] Registered new tserver with Master: 1aec0d3c40b045acbc52248ebc553005 (127.30.16.65:33547)
I20260927 07:11:29.655107 18206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.65:55067
W20260927 07:11:29.983893 18396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:29.984294 18396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:29.984445 18396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:29.991909 18396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:29.992069 18396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:29.992249 18396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.66
I20260927 07:11:30.001890 18396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.16.66
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.66
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:30.003333 18396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:30.005239 18396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:30.019471 18401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:30.019476 18404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:30.019524 18402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:30.020174 18396 server_base.cc:1034] running on GCE node
I20260927 07:11:30.020865 18396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:30.021962 18396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:30.023204 18396 hybrid_clock.cc:648] HybridClock initialized: now 1790493090023137 us; error 63 us; skew 500 ppm
I20260927 07:11:30.023556 18396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:30.026068 18396 webserver.cc:460] Webserver started at http://127.30.16.66:40267/ using document root <none> and password file <none>
I20260927 07:11:30.026922 18396 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:30.027057 18396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:30.027453 18396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:30.030486 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data/instance:
uuid: "621938ebcdf64b87bc01d352f60658de"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "ad8d5c11b5d85b5836f6bf6198a6dd2e"
server_key_iv: "3cb09dfbc34c5238896ccf9845e28c6d"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.031409 18396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance:
uuid: "621938ebcdf64b87bc01d352f60658de"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "ad8d5c11b5d85b5836f6bf6198a6dd2e"
server_key_iv: "3cb09dfbc34c5238896ccf9845e28c6d"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.037034 18396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:30.040865 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.042253 18396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 07:11:30.042464 18396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal
uuid: "621938ebcdf64b87bc01d352f60658de"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "ad8d5c11b5d85b5836f6bf6198a6dd2e"
server_key_iv: "3cb09dfbc34c5238896ccf9845e28c6d"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.042759 18396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:30.060449 18396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:30.061327 18396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:30.061653 18396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:30.062707 18396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:30.064567 18396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:30.064679 18396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.064791 18396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:30.064873 18396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.102221 18396 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.66:35185
I20260927 07:11:30.102274 18522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.66:35185 every 8 connection(s)
I20260927 07:11:30.103690 18396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/data/info.pb
I20260927 07:11:30.108979 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18396
I20260927 07:11:30.109167 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal/instance
I20260927 07:11:30.109601 30785 external_mini_cluster.cc:1428] Setting key 87a7763b9ff271721cdc954bb28cf704
I20260927 07:11:30.112926 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.16.67:0
--local_ip_for_outbound_sockets=127.30.16.67
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--builtin_ntp_servers=127.30.16.84:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 07:11:30.118439 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.106308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:54671 (local address 127.30.16.126:40497)
0927 07:11:30.106518 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.106523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.108344 (+ 1821us) server_negotiation.cc:407] Connection header received
0927 07:11:30.110723 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.110728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.110799 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.110912 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:30.112578 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.113493 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.115352 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.115791 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.115944 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.117480 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:30.117486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:30.117490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:30.117511 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:30.117543 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:30.117555 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:30.117653 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.117919 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.118053 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 07:11:30.119731 18525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.106953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:40497 (local address 127.30.16.66:54671)
0927 07:11:30.108165 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.108216 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.110372 (+ 2156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.111356 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.111370 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.111508 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:30.112385 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.112405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.113663 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.113669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.115179 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.115191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.116039 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.116046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.116306 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.117103 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:30.117106 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:30.117151 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:30.117241 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:30.117671 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:30.117684 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:30.117700 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.117909 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.118443 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":426,"threads_started":1}
I20260927 07:11:30.121400 18523 heartbeater.cc:347] Connected to a master server at 127.30.16.126:40497
I20260927 07:11:30.121727 18523 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:30.122605 18523 heartbeater.cc:511] Master 127.30.16.126:40497 requested a full tablet report, sending...
I20260927 07:11:30.124094 18206 ts_manager.cc:194] Registered new tserver with Master: 621938ebcdf64b87bc01d352f60658de (127.30.16.66:35185)
I20260927 07:11:30.124917 18206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.66:54671
W20260927 07:11:30.459548 18526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:30.459887 18526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:30.460059 18526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:30.467571 18526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:30.467739 18526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:30.467931 18526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.16.67
I20260927 07:11:30.477551 18526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.16.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.16.67
--webserver_port=0
--tserver_master_addrs=127.30.16.126:40497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.16.67
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:30.478981 18526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:30.480796 18526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:30.494642 18532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:30.494694 18531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:30.494671 18534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:30.495407 18526 server_base.cc:1034] running on GCE node
I20260927 07:11:30.496083 18526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:30.497099 18526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:30.498377 18526 hybrid_clock.cc:648] HybridClock initialized: now 1790493090498311 us; error 43 us; skew 500 ppm
I20260927 07:11:30.498781 18526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 07:11:30.501340 18526 webserver.cc:460] Webserver started at http://127.30.16.67:35743/ using document root <none> and password file <none>
I20260927 07:11:30.502099 18526 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:30.502236 18526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:30.502619 18526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:30.505653 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data/instance:
uuid: "aa7bba5ab19640aea605173b5830adbd"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "1411615bbb0e0e539f2ce74b73588eff"
server_key_iv: "03c9aefc6fc70929bc8accf290088695"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.506508 18526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance:
uuid: "aa7bba5ab19640aea605173b5830adbd"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "1411615bbb0e0e539f2ce74b73588eff"
server_key_iv: "03c9aefc6fc70929bc8accf290088695"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.512169 18526 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:11:30.515790 18540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.517196 18526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:30.517379 18526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal
uuid: "aa7bba5ab19640aea605173b5830adbd"
format_stamp: "Formatted at 2026-09-27 07:11:30 on dist-test-slave-8vqf"
server_key: "1411615bbb0e0e539f2ce74b73588eff"
server_key_iv: "03c9aefc6fc70929bc8accf290088695"
server_key_version: "encryptionkey@0"
I20260927 07:11:30.517611 18526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:30.539772 18526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:30.540830 18526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:30.541224 18526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:30.542657 18526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 07:11:30.544430 18526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 07:11:30.544546 18526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.544669 18526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 07:11:30.544765 18526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:30.578603 18526 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.67:43383
I20260927 07:11:30.578719 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.67:43383 every 8 connection(s)
I20260927 07:11:30.580022 18526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/data/info.pb
I20260927 07:11:30.590341 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18526
I20260927 07:11:30.590482 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal/instance
I20260927 07:11:30.591136 30785 external_mini_cluster.cc:1428] Setting key 3e3b4b7191242479b506cd615972a4d5
I20260927 07:11:30.593380 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.582605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.67:34605 (local address 127.30.16.126:40497)
0927 07:11:30.582789 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.582793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.584428 (+ 1635us) server_negotiation.cc:407] Connection header received
0927 07:11:30.586304 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.586310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.586406 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.586517 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:30.587714 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.588472 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.590087 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.590658 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.590769 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.592494 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:30.592499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:30.592502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:30.592540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:30.592581 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:30.592600 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:30.592700 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.592981 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.593085 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:11:30.594756 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.583197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:40497 (local address 127.30.16.67:34605)
0927 07:11:30.584251 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.584289 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.586019 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.586843 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.586853 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.586949 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:11:30.587540 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.587557 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.588638 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.588642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.589897 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.589936 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.590892 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.590899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.591217 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.592085 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 07:11:30.592096 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:11:30.592157 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 07:11:30.592278 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:11:30.592716 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:11:30.592727 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:11:30.592735 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.592954 (+ 219us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.593411 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":319,"threads_started":1}
I20260927 07:11:30.596086 18653 heartbeater.cc:347] Connected to a master server at 127.30.16.126:40497
I20260927 07:11:30.596421 18653 heartbeater.cc:464] Registering TS with master...
I20260927 07:11:30.597257 18653 heartbeater.cc:511] Master 127.30.16.126:40497 requested a full tablet report, sending...
I20260927 07:11:30.598968 18206 ts_manager.cc:194] Registered new tserver with Master: aa7bba5ab19640aea605173b5830adbd (127.30.16.67:43383)
I20260927 07:11:30.599941 18206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.16.67:34605
I20260927 07:11:30.606899 30785 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 07:11:30.616936 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.611089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.30.16.126:40497)
0927 07:11:30.611328 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.611338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.611662 (+ 324us) server_negotiation.cc:407] Connection header received
0927 07:11:30.611806 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.611810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.611892 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.611975 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:30.612448 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.613146 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.614469 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.615000 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.615149 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.616181 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:30.616191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:30.616194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:30.616214 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:30.616246 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:30.616258 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:30.616359 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.616522 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.616632 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:11:30.620951 18206 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 07:11:30.628239 18206 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 07:11:30.657398 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.641400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46406 (local address 127.30.16.65:33547)
0927 07:11:30.647771 (+ 6371us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.647782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.648577 (+ 795us) server_negotiation.cc:407] Connection header received
0927 07:11:30.648977 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.648987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.649250 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.649446 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:30.650632 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.652084 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.654777 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.656158 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.656303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.656598 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.656815 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.657009 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6084,"thread_start_us":5933,"threads_started":1}
I20260927 07:11:30.657404 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.641127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:33547 (local address 127.0.0.1:46406)
0927 07:11:30.648418 (+ 7291us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.648462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.648675 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.649741 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.649749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.649780 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:30.650409 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.650430 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.652232 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.652238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.654594 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.654619 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.656358 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.656365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.656558 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.656613 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.656752 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.656962 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6797,"mutex_wait_us":83,"spinlock_wait_cycles":17152,"thread_start_us":6610,"threads_started":1}
I20260927 07:11:30.658032 18666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.641296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.66:35185 (local address 127.0.0.1:44738)
0927 07:11:30.648418 (+ 7122us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.648462 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.648676 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.649774 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.649782 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.649826 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:30.650924 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.650938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.652685 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.652691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.655056 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.655071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.657211 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.657225 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.657458 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.657497 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.657583 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.657723 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6833,"mutex_wait_us":35,"spinlock_wait_cycles":7168,"thread_start_us":6680,"threads_started":1}
I20260927 07:11:30.658135 18392 heartbeater.cc:503] Master 127.30.16.126:40497 was elected leader, sending a full tablet report...
I20260927 07:11:30.658918 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.642085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44738 (local address 127.30.16.66:35185)
0927 07:11:30.647136 (+ 5051us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.647150 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.648654 (+ 1504us) server_negotiation.cc:407] Connection header received
0927 07:11:30.648979 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.648989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.649256 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.649474 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:30.651090 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.652543 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.655259 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.656699 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.656847 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.657078 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.658318 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.658522 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4828,"thread_start_us":4674,"threads_started":1}
I20260927 07:11:30.659742 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.650013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:43383 (local address 127.0.0.1:43078)
0927 07:11:30.650708 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.650731 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.650880 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.651496 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.651500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.651525 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:30.651868 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.651878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.653628 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.653636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.655693 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.655703 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.659183 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.659191 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.659329 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.659354 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.659418 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.659506 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":38,"thread_start_us":363,"threads_started":1}
I20260927 07:11:30.660468 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.649091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.30.16.67:43383)
0927 07:11:30.650417 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.650427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.650843 (+ 416us) server_negotiation.cc:407] Connection header received
0927 07:11:30.651058 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.651066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.651299 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.651480 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:30.652041 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.653456 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.655875 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.657087 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.657214 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.657498 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.660020 (+ 2522us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.660202 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":939,"threads_started":1}
I20260927 07:11:30.661759 18327 tablet_service.cc:1467] Processing CreateTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (DEFAULT_TABLE table=test-table [id=57598dc344aa47408e2bfe0c7f2094eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:30.663132 18327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:30.663614 18588 tablet_service.cc:1467] Processing CreateTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (DEFAULT_TABLE table=test-table [id=57598dc344aa47408e2bfe0c7f2094eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:30.664990 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:30.668270 18458 tablet_service.cc:1467] Processing CreateTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (DEFAULT_TABLE table=test-table [id=57598dc344aa47408e2bfe0c7f2094eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 07:11:30.669734 18458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:30.675135 18671 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Bootstrap starting.
I20260927 07:11:30.678386 18671 tablet_bootstrap.cc:654] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:30.679435 18671 log.cc:824] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:30.680447 18672 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Bootstrap starting.
I20260927 07:11:30.684806 18674 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Bootstrap starting.
I20260927 07:11:30.685263 18672 tablet_bootstrap.cc:654] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:30.686517 18672 log.cc:824] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:30.688272 18674 tablet_bootstrap.cc:654] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:30.688794 18671 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: No bootstrap required, opened a new log
I20260927 07:11:30.689136 18671 ts_tablet_manager.cc:1397] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260927 07:11:30.689551 18674 log.cc:824] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:30.691551 18672 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: No bootstrap required, opened a new log
I20260927 07:11:30.691854 18672 ts_tablet_manager.cc:1397] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260927 07:11:30.692612 18674 tablet_bootstrap.cc:492] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: No bootstrap required, opened a new log
I20260927 07:11:30.692860 18674 ts_tablet_manager.cc:1397] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 07:11:30.697209 18671 raft_consensus.cc:348] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.697615 18671 raft_consensus.cc:374] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:30.697711 18671 raft_consensus.cc:729] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aec0d3c40b045acbc52248ebc553005, State: Initialized, Role: FOLLOWER
I20260927 07:11:30.698374 18671 consensus_queue.cc:260] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.700295 18671 ts_tablet_manager.cc:1428] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260927 07:11:30.703223 18672 raft_consensus.cc:348] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.703723 18672 raft_consensus.cc:374] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:30.703850 18672 raft_consensus.cc:729] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa7bba5ab19640aea605173b5830adbd, State: Initialized, Role: FOLLOWER
I20260927 07:11:30.703760 18674 raft_consensus.cc:348] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.704324 18674 raft_consensus.cc:374] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:30.704439 18674 raft_consensus.cc:729] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621938ebcdf64b87bc01d352f60658de, State: Initialized, Role: FOLLOWER
I20260927 07:11:30.704519 18672 consensus_queue.cc:260] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.705035 18674 consensus_queue.cc:260] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.706136 18653 heartbeater.cc:503] Master 127.30.16.126:40497 was elected leader, sending a full tablet report...
I20260927 07:11:30.707228 18672 ts_tablet_manager.cc:1428] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260927 07:11:30.708153 18523 heartbeater.cc:503] Master 127.30.16.126:40497 was elected leader, sending a full tablet report...
I20260927 07:11:30.708848 18674 ts_tablet_manager.cc:1428] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260927 07:11:30.831748 18654 tablet.cc:2369] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:30.854951 18679 raft_consensus.cc:482] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 07:11:30.855283 18679 raft_consensus.cc:504] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
W20260927 07:11:30.855955 18524 tablet.cc:2369] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:30.857409 18679 leader_election.cc:288] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aec0d3c40b045acbc52248ebc553005 (127.30.16.65:33547), aa7bba5ab19640aea605173b5830adbd (127.30.16.67:43383)
I20260927 07:11:30.864375 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.857940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:43609 (local address 127.30.16.65:33547)
0927 07:11:30.858143 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.858150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.858851 (+ 701us) server_negotiation.cc:407] Connection header received
0927 07:11:30.859033 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.859039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.859137 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.859222 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:30.859866 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.860597 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.862665 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.863443 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.863634 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.863705 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.864021 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.864150 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:11:30.864387 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.857838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.30.16.66:38857 (local address 127.30.16.67:43383)
0927 07:11:30.858060 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.858069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.859024 (+ 955us) server_negotiation.cc:407] Connection header received
0927 07:11:30.859166 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.859169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.859261 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.859377 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 07:11:30.859952 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.860641 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.862634 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.863416 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.863570 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.863652 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.863980 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.864115 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 07:11:30.864491 18681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.857693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.67:43383 (local address 127.30.16.66:38857)
0927 07:11:30.858870 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.858888 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.859041 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.859397 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.859401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.859432 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:30.859799 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.859812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.860799 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.860802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.862416 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.862431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.863612 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.863620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.863802 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.863826 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.863902 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.864101 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"mutex_wait_us":118,"thread_start_us":574,"threads_started":1}
I20260927 07:11:30.864689 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.857630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.65:33547 (local address 127.30.16.66:43609)
0927 07:11:30.858730 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0927 07:11:30.858746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 07:11:30.858892 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:11:30.859267 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:11:30.859272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:11:30.859298 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 07:11:30.859714 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.859725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.860728 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.860735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.862442 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:11:30.862453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.863660 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.863669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:11:30.863830 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.863851 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 07:11:30.863965 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 07:11:30.864129 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":989,"mutex_wait_us":104,"thread_start_us":604,"threads_started":1}
I20260927 07:11:30.865554 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67972c1d0ec4d6c9a9bdd21ea0376fb" candidate_uuid: "621938ebcdf64b87bc01d352f60658de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7bba5ab19640aea605173b5830adbd" is_pre_election: true
I20260927 07:11:30.865717 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67972c1d0ec4d6c9a9bdd21ea0376fb" candidate_uuid: "621938ebcdf64b87bc01d352f60658de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aec0d3c40b045acbc52248ebc553005" is_pre_election: true
I20260927 07:11:30.866099 18608 raft_consensus.cc:2454] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621938ebcdf64b87bc01d352f60658de in term 0.
I20260927 07:11:30.866279 18347 raft_consensus.cc:2454] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 621938ebcdf64b87bc01d352f60658de in term 0.
I20260927 07:11:30.866832 18411 leader_election.cc:302] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621938ebcdf64b87bc01d352f60658de, aa7bba5ab19640aea605173b5830adbd; no voters:
I20260927 07:11:30.867239 18679 raft_consensus.cc:2781] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 07:11:30.867383 18679 raft_consensus.cc:482] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 07:11:30.867482 18679 raft_consensus.cc:3037] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:30.869639 18679 raft_consensus.cc:504] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.870304 18679 leader_election.cc:288] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [CANDIDATE]: Term 1 election: Requested vote from peers 1aec0d3c40b045acbc52248ebc553005 (127.30.16.65:33547), aa7bba5ab19640aea605173b5830adbd (127.30.16.67:43383)
I20260927 07:11:30.870733 18347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67972c1d0ec4d6c9a9bdd21ea0376fb" candidate_uuid: "621938ebcdf64b87bc01d352f60658de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aec0d3c40b045acbc52248ebc553005"
I20260927 07:11:30.870877 18608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d67972c1d0ec4d6c9a9bdd21ea0376fb" candidate_uuid: "621938ebcdf64b87bc01d352f60658de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa7bba5ab19640aea605173b5830adbd"
I20260927 07:11:30.870965 18347 raft_consensus.cc:3037] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:30.871085 18608 raft_consensus.cc:3037] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:30.874235 18608 raft_consensus.cc:2454] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621938ebcdf64b87bc01d352f60658de in term 1.
I20260927 07:11:30.874230 18347 raft_consensus.cc:2454] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 621938ebcdf64b87bc01d352f60658de in term 1.
I20260927 07:11:30.874657 18411 leader_election.cc:302] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 621938ebcdf64b87bc01d352f60658de, aa7bba5ab19640aea605173b5830adbd; no voters:
I20260927 07:11:30.874961 18679 raft_consensus.cc:2781] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:30.875407 18679 raft_consensus.cc:686] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 1 LEADER]: Becoming Leader. State: Replica: 621938ebcdf64b87bc01d352f60658de, State: Running, Role: LEADER
I20260927 07:11:30.875900 18679 consensus_queue.cc:237] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } }
I20260927 07:11:30.880647 18205 catalog_manager.cc:5495] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de reported cstate change: term changed from 0 to 1, leader changed from <none> to 621938ebcdf64b87bc01d352f60658de (127.30.16.66). New cstate: current_term: 1 leader_uuid: "621938ebcdf64b87bc01d352f60658de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "621938ebcdf64b87bc01d352f60658de" member_type: VOTER last_known_addr { host: "127.30.16.66" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 } health_report { overall_health: UNKNOWN } } }
W20260927 07:11:30.884042 18393 tablet.cc:2369] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:11:30.949541 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.943945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44740 (local address 127.30.16.66:35185)
0927 07:11:30.944191 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.944195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:30.944231 (+ 36us) server_negotiation.cc:407] Connection header received
0927 07:11:30.944424 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:30.944428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:30.944525 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:30.944648 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:11:30.944952 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.945650 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.947360 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:30.947806 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:30.947914 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:30.948412 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:11:30.948957 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:11:30.949077 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:30.949151 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 07:11:30.949219 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 07:11:30.958240 18347 raft_consensus.cc:1261] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 1 FOLLOWER]: Refusing update from remote peer 621938ebcdf64b87bc01d352f60658de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:30.958235 18608 raft_consensus.cc:1261] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 1 FOLLOWER]: Refusing update from remote peer 621938ebcdf64b87bc01d352f60658de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:11:30.959309 18682 consensus_queue.cc:1035] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7bba5ab19640aea605173b5830adbd" member_type: VOTER last_known_addr { host: "127.30.16.67" port: 43383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:30.959700 18679 consensus_queue.cc:1035] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aec0d3c40b045acbc52248ebc553005" member_type: VOTER last_known_addr { host: "127.30.16.65" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:11:30.990131 18206 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:30.990459 18206 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 07:11:30.994189 18206 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8645ae865b474159951abad8555b0f06: Sending DeleteTablet for 3 replicas of tablet d67972c1d0ec4d6c9a9bdd21ea0376fb
I20260927 07:11:30.995492 18327 tablet_service.cc:1514] Processing DeleteTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:30 UTC) from {username='slave'} at 127.0.0.1:46406
I20260927 07:11:30.995572 18458 tablet_service.cc:1514] Processing DeleteTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:30 UTC) from {username='slave'} at 127.0.0.1:44738
I20260927 07:11:30.996284 18588 tablet_service.cc:1514] Processing DeleteTablet for tablet d67972c1d0ec4d6c9a9bdd21ea0376fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 07:11:30 UTC) from {username='slave'} at 127.0.0.1:43078
I20260927 07:11:30.997083 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18265
I20260927 07:11:31.010054 18696 tablet_replica.cc:330] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: stopping tablet replica
I20260927 07:11:31.011806 18696 raft_consensus.cc:2229] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 1 LEADER]: Raft consensus shutting down.
I20260927 07:11:31.013319 18696 raft_consensus.cc:2258] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:31.014340 18695 tablet_replica.cc:330] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: stopping tablet replica
I20260927 07:11:31.014914 18695 raft_consensus.cc:2229] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:31.015894 18695 raft_consensus.cc:2258] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:31.016186 18696 ts_tablet_manager.cc:1905] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:31.018199 18695 ts_tablet_manager.cc:1905] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:31.024411 18696 ts_tablet_manager.cc:1918] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:31.024619 18696 log.cc:1196] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-1/wal/wals/d67972c1d0ec4d6c9a9bdd21ea0376fb
I20260927 07:11:31.024768 18697 tablet_replica.cc:330] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: stopping tablet replica
I20260927 07:11:31.025211 18696 ts_tablet_manager.cc:1939] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 621938ebcdf64b87bc01d352f60658de: Deleting consensus metadata
I20260927 07:11:31.025244 18697 raft_consensus.cc:2229] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 07:11:31.025815 18697 raft_consensus.cc:2258] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 07:11:31.026109 18695 ts_tablet_manager.cc:1918] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:31.026358 18695 log.cc:1196] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-0/wal/wals/d67972c1d0ec4d6c9a9bdd21ea0376fb
I20260927 07:11:31.026849 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:30.999163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.30.16.65:33547)
0927 07:11:30.999369 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:30.999375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:31.011896 (+ 12521us) server_negotiation.cc:407] Connection header received
0927 07:11:31.012953 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:31.012962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:31.013087 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:31.013179 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:31.014384 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.015661 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.019688 (+ 4027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.020407 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.020515 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:31.023188 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:31.023201 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:31.023220 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:31.024101 (+ 881us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:31.024169 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:31.024198 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:31.024299 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:31.026246 (+ 1947us) server_negotiation.cc:299] Negotiation successful
0927 07:11:31.026409 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:11:31.027318 18695 ts_tablet_manager.cc:1939] T d67972c1d0ec4d6c9a9bdd21ea0376fb P 1aec0d3c40b045acbc52248ebc553005: Deleting consensus metadata
I20260927 07:11:31.027469 18190 catalog_manager.cc:4841] TS 621938ebcdf64b87bc01d352f60658de (127.30.16.66:35185): tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (table test-table [id=57598dc344aa47408e2bfe0c7f2094eb]) successfully deleted
I20260927 07:11:31.028344 18387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:31.030359 18697 ts_tablet_manager.cc:1905] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 07:11:31.037709 18697 ts_tablet_manager.cc:1918] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 07:11:31.037905 18697 log.cc:1196] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Deleting WAL directory at /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790492767955610-30785-0/minicluster-data/ts-2/wal/wals/d67972c1d0ec4d6c9a9bdd21ea0376fb
I20260927 07:11:31.038470 18697 ts_tablet_manager.cc:1939] T d67972c1d0ec4d6c9a9bdd21ea0376fb P aa7bba5ab19640aea605173b5830adbd: Deleting consensus metadata
I20260927 07:11:31.040647 18190 catalog_manager.cc:4841] TS aa7bba5ab19640aea605173b5830adbd (127.30.16.67:43383): tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (table test-table [id=57598dc344aa47408e2bfe0c7f2094eb]) successfully deleted
I20260927 07:11:31.130440 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18265
I20260927 07:11:31.131132 18192 catalog_manager.cc:4841] TS 1aec0d3c40b045acbc52248ebc553005 (127.30.16.65:33547): tablet d67972c1d0ec4d6c9a9bdd21ea0376fb (table test-table [id=57598dc344aa47408e2bfe0c7f2094eb]) successfully deleted
I20260927 07:11:31.151640 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18396
I20260927 07:11:31.159024 18668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:31.152727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.30.16.66:35185)
0927 07:11:31.152960 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:31.152965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:31.152980 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:11:31.153036 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:31.153040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:31.153136 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:31.153259 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:31.153664 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.154455 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.155717 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.156155 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.156299 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:31.157405 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:31.157414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:31.157433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:31.158211 (+ 778us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:31.158265 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:31.158285 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:31.158415 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:31.158570 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:11:31.158674 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:11:31.159859 18518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:31.271925 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18396
I20260927 07:11:31.291342 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18526
I20260927 07:11:31.297626 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:31.292103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.30.16.67:43383)
0927 07:11:31.292265 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:31.292269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:31.292287 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:11:31.292434 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:31.292437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:31.292517 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:31.292610 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:31.292944 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.293641 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.294955 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:31.295354 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:31.295448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:31.296396 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:31.296405 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:31.296420 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:31.296998 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:31.297045 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:31.297063 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:31.297157 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:31.297263 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 07:11:31.297362 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 07:11:31.298300 18648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:31.387827 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18526
I20260927 07:11:31.405798 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18175
I20260927 07:11:31.406927 18236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:31.518277 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18175
2026-09-27T07:11:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2966 ms)
[----------] 6 tests from EncryptionPolicyTest (23104 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:31 dist-test-slave-8vqf krb5kdc[18705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:31 dist-test-slave-8vqf krb5kdc[18705](info): set up 2 sockets
Sep 27 07:11:31 dist-test-slave-8vqf krb5kdc[18705](info): commencing operation
krb5kdc: starting...
W20260927 07:11:32.667058 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:32 dist-test-slave-8vqf krb5kdc[18705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:32.935550 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:33.277673 18718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:33.278067 18718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:33.278234 18718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:33.285816 18718 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:33.286010 18718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:33.286128 18718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:33.286239 18718 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:33.286329 18718 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:33.296533 18718 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:33.298008 18718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:33.299465 18718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:33.311584 18726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:33.311780 18723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:33.312018 18724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:33.313012 18718 server_base.cc:1034] running on GCE node
I20260927 07:11:33.313925 18718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:33.315658 18718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:33.316875 18718 hybrid_clock.cc:648] HybridClock initialized: now 1790493093316835 us; error 31 us; skew 500 ppm
I20260927 07:11:33.317261 18718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:33 dist-test-slave-8vqf krb5kdc[18705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:33.320637 18718 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:33.322227 18718 webserver.cc:460] Webserver started at http://127.0.0.1:46641/ using document root <none> and password file <none>
I20260927 07:11:33.323071 18718 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:33.323235 18718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:33.323593 18718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:33.327184 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "0c76762a4a154fa49b13aeed594b731c"
format_stamp: "Formatted at 2026-09-27 07:11:33 on dist-test-slave-8vqf"
server_key: "a5c3f6db2e02a3cf096a0619faa57dd5"
server_key_iv: "375a971e0604a77f1d1124c73fc7956b"
server_key_version: "encryptionkey@0"
I20260927 07:11:33.328195 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "0c76762a4a154fa49b13aeed594b731c"
format_stamp: "Formatted at 2026-09-27 07:11:33 on dist-test-slave-8vqf"
server_key: "a5c3f6db2e02a3cf096a0619faa57dd5"
server_key_iv: "375a971e0604a77f1d1124c73fc7956b"
server_key_version: "encryptionkey@0"
I20260927 07:11:33.334216 18718 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260927 07:11:33.338413 18733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:33.339918 18718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:33.340111 18718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "0c76762a4a154fa49b13aeed594b731c"
format_stamp: "Formatted at 2026-09-27 07:11:33 on dist-test-slave-8vqf"
server_key: "a5c3f6db2e02a3cf096a0619faa57dd5"
server_key_iv: "375a971e0604a77f1d1124c73fc7956b"
server_key_version: "encryptionkey@0"
I20260927 07:11:33.340334 18718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:33.359472 18718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:33.364037 18718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:33.364384 18718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:33.383733 18718 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41937
I20260927 07:11:33.383752 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41937 every 8 connection(s)
I20260927 07:11:33.385247 18718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:33.389204 18785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:33.391620 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18718
I20260927 07:11:33.391855 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:33.392520 30785 external_mini_cluster.cc:1428] Setting key 8fe9dcf1042889e523402c33d08f57ff
I20260927 07:11:33.403184 18785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c: Bootstrap starting.
I20260927 07:11:33.406975 18785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:33.408000 18785 log.cc:824] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:11:33 dist-test-slave-8vqf krb5kdc[18705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:11:33.411382 18785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c: No bootstrap required, opened a new log
I20260927 07:11:33.418267 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:33.394695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43123 (local address 127.0.0.1:41937)
0927 07:11:33.396339 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:33.396355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:33.396406 (+ 51us) server_negotiation.cc:407] Connection header received
0927 07:11:33.398848 (+ 2442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:33.398881 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:33.399464 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:33.399942 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:33.401290 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:33.403351 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:33.404639 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:33.405727 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:33.405908 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:33.410436 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:33.410444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:33.410460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:33.410500 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:33.413731 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:33.414764 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:33.414771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:33.414777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:33.414965 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:33.415477 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:33.415480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:33.415482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:33.416009 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:33.416251 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:33.416764 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 07:11:33.417130 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":424,"threads_started":1}
I20260927 07:11:33.420517 18785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260927 07:11:33.420975 18785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:33.421095 18785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c76762a4a154fa49b13aeed594b731c, State: Initialized, Role: FOLLOWER
I20260927 07:11:33.421816 18785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260927 07:11:33.422101 18785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:33.422242 18785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:33.422457 18785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:33.422724 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:33.425800 18785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260927 07:11:33.426471 18785 leader_election.cc:302] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c76762a4a154fa49b13aeed594b731c; no voters:
I20260927 07:11:33.427284 18785 leader_election.cc:288] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:33.427425 18793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:33.429790 18785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:33.429872 18793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [term 1 LEADER]: Becoming Leader. State: Replica: 0c76762a4a154fa49b13aeed594b731c, State: Running, Role: LEADER
I20260927 07:11:33.430673 18793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } }
I20260927 07:11:33.434269 18796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c76762a4a154fa49b13aeed594b731c. Latest consensus state: current_term: 1 leader_uuid: "0c76762a4a154fa49b13aeed594b731c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260927 07:11:33.434320 18795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c76762a4a154fa49b13aeed594b731c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c76762a4a154fa49b13aeed594b731c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41937 } } }
I20260927 07:11:33.434767 18796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:33.434806 18795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:33.436371 18800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:33.442394 18800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:33.450690 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:33.429173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36527 (local address 127.0.0.1:41937)
0927 07:11:33.429370 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:33.429376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:33.429633 (+ 257us) server_negotiation.cc:407] Connection header received
0927 07:11:33.430024 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:33.430033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:33.430124 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:33.430198 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:33.432255 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:33.433964 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:33.436206 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:33.437152 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:33.437291 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:33.441538 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:33.441545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:33.441550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:33.441590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:33.445692 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:33.446800 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:33.446805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:33.446808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:33.446958 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:33.447554 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:33.447559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:33.447562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:33.447958 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:33.448205 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:33.449921 (+ 1716us) server_negotiation.cc:299] Negotiation successful
0927 07:11:33.450115 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:11:33.457856 18800 catalog_manager.cc:1347] Generated new cluster ID: 386cb188e6534f8aac4601e9a023dc65
I20260927 07:11:33.457995 18800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:33.469671 18800 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:33.471318 18800 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:33.490376 18800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c76762a4a154fa49b13aeed594b731c: Generated new TSK 0
I20260927 07:11:33.491372 18800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:33.500432 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18718
I20260927 07:11:33.502759 18780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:33.596050 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18718
W20260927 07:11:33.613713 18703 connection.cc:535] client connection to 127.0.0.1:41937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41937): Connection reset by peer (error 104)
2026-09-27T07:11:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2072 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:33 dist-test-slave-8vqf krb5kdc[18817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:33 dist-test-slave-8vqf krb5kdc[18817](info): set up 2 sockets
Sep 27 07:11:33 dist-test-slave-8vqf krb5kdc[18817](info): commencing operation
krb5kdc: starting...
W20260927 07:11:34.703008 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:34 dist-test-slave-8vqf krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:34.962940 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:35.305176 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:35.305572 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:35.305744 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:35.313367 18830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:35.313503 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:35.313624 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:35.313745 18830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:35.313829 18830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:35.324497 18830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:35.325932 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:35.327387 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:35.340080 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:35.340497 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:35.340530 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:35.341019 18830 server_base.cc:1034] running on GCE node
I20260927 07:11:35.341881 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:35.343513 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:35.344749 18830 hybrid_clock.cc:648] HybridClock initialized: now 1790493095344704 us; error 36 us; skew 500 ppm
I20260927 07:11:35.345126 18830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:35 dist-test-slave-8vqf krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:35.354493 18830 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:35.356280 18830 webserver.cc:460] Webserver started at http://127.0.0.1:45137/ using document root <none> and password file <none>
I20260927 07:11:35.357036 18830 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:35.357191 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:35.357563 18830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:35.360877 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294"
format_stamp: "Formatted at 2026-09-27 07:11:35 on dist-test-slave-8vqf"
server_key: "2a9d8b268763a19c6f1d459c3732ffad"
server_key_iv: "7f00ee19b367ed421a12c7886fbf8edd"
server_key_version: "encryptionkey@0"
I20260927 07:11:35.361727 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294"
format_stamp: "Formatted at 2026-09-27 07:11:35 on dist-test-slave-8vqf"
server_key: "2a9d8b268763a19c6f1d459c3732ffad"
server_key_iv: "7f00ee19b367ed421a12c7886fbf8edd"
server_key_version: "encryptionkey@0"
I20260927 07:11:35.367429 18830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:11:35.371127 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:35.372612 18830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 07:11:35.372817 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294"
format_stamp: "Formatted at 2026-09-27 07:11:35 on dist-test-slave-8vqf"
server_key: "2a9d8b268763a19c6f1d459c3732ffad"
server_key_iv: "7f00ee19b367ed421a12c7886fbf8edd"
server_key_version: "encryptionkey@0"
I20260927 07:11:35.373034 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:35.385489 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:35.389742 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:35.390054 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:35.410295 18830 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33115
I20260927 07:11:35.410315 18896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33115 every 8 connection(s)
I20260927 07:11:35.411952 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:35.416157 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:35.419399 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18830
I20260927 07:11:35.419592 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:35.420135 30785 external_mini_cluster.cc:1428] Setting key 00b7a10cad498bb645376fb61d18d587
I20260927 07:11:35.428730 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294: Bootstrap starting.
I20260927 07:11:35.432356 18897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:35.433761 18897 log.cc:824] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:35.434443 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:35.422115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59411 (local address 127.0.0.1:33115)
0927 07:11:35.423019 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:35.423032 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:35.423074 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:11:35.424793 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:35.424829 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:35.425258 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:35.425695 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:35.426294 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:35.427712 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:35.429225 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:35.429934 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:35.430137 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:35.431493 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:35.431501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:35.431516 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:35.432140 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:35.432221 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:35.432249 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:35.432428 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:35.432776 (+ 348us) server_negotiation.cc:299] Negotiation successful
0927 07:11:35.432986 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":447,"threads_started":1}
I20260927 07:11:35.437242 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294: No bootstrap required, opened a new log
I20260927 07:11:35.438755 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:35.447995 18897 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260927 07:11:35.448592 18897 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:35.448722 18897 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d5f86f2bd24e2eadb9dbc1aaccf294, State: Initialized, Role: FOLLOWER
I20260927 07:11:35.449440 18897 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260927 07:11:35.449733 18897 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:35.449889 18897 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:35.450100 18897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:35.451201 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:35.444160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34203 (local address 127.0.0.1:33115)
0927 07:11:35.444360 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:35.444366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:35.444734 (+ 368us) server_negotiation.cc:407] Connection header received
0927 07:11:35.444905 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:35.444914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:35.445005 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:35.445132 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:35.445646 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:35.446881 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:35.448223 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:35.448920 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:35.449051 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:35.450288 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:35.450294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:35.450298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:35.450320 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:35.450389 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:35.450415 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:35.450585 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:35.450788 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 07:11:35.450849 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 07:11:35.452960 18897 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260927 07:11:35.453491 18897 leader_election.cc:302] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8d5f86f2bd24e2eadb9dbc1aaccf294; no voters:
I20260927 07:11:35.454080 18897 leader_election.cc:288] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:35.454188 18907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:35.455013 18907 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [term 1 LEADER]: Becoming Leader. State: Replica: e8d5f86f2bd24e2eadb9dbc1aaccf294, State: Running, Role: LEADER
I20260927 07:11:35.455601 18907 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260927 07:11:35.456420 18897 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:35.458688 18908 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260927 07:11:35.459163 18908 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:35.460168 18914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:35.459959 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8d5f86f2bd24e2eadb9dbc1aaccf294. Latest consensus state: current_term: 1 leader_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8d5f86f2bd24e2eadb9dbc1aaccf294" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260927 07:11:35.461369 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:35.465987 18914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:35.475803 18914 catalog_manager.cc:1347] Generated new cluster ID: 1d5f7ff188ab45de8370444cce2e6eba
I20260927 07:11:35.475955 18914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:35.487703 18914 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:35.489045 18914 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:35.495349 18914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8d5f86f2bd24e2eadb9dbc1aaccf294: Generated new TSK 0
I20260927 07:11:35.496206 18914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:35.505909 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18830
W20260927 07:11:35.507181 18892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59411
W20260927 07:11:35.508061 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18830: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:35.508253 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18830
W20260927 07:11:35.525429 18815 connection.cc:535] client connection to 127.0.0.1:33115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33115): Connection reset by peer (error 104)
2026-09-27T07:11:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:35 dist-test-slave-8vqf krb5kdc[18928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:35 dist-test-slave-8vqf krb5kdc[18928](info): set up 2 sockets
Sep 27 07:11:35 dist-test-slave-8vqf krb5kdc[18928](info): commencing operation
krb5kdc: starting...
W20260927 07:11:36.639145 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:36 dist-test-slave-8vqf krb5kdc[18928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:36.901865 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:37.242195 18941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:37.242499 18941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:37.242700 18941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:37.250139 18941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:37.250257 18941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:37.250393 18941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:37.250502 18941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:37.250608 18941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:37.261338 18941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:37.262725 18941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:37.264079 18941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:37.276052 18947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:37.276127 18946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:37.276099 18949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:37.277148 18941 server_base.cc:1034] running on GCE node
I20260927 07:11:37.277912 18941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:37.279413 18941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:37.280620 18941 hybrid_clock.cc:648] HybridClock initialized: now 1790493097280581 us; error 29 us; skew 500 ppm
I20260927 07:11:37.280951 18941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:37 dist-test-slave-8vqf krb5kdc[18928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:37.284071 18941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:37.285622 18941 webserver.cc:460] Webserver started at http://127.0.0.1:46249/ using document root <none> and password file <none>
I20260927 07:11:37.286386 18941 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:37.286491 18941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:37.286839 18941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:37.289903 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "387bbe581a214b7fbc242efa489ea382"
format_stamp: "Formatted at 2026-09-27 07:11:37 on dist-test-slave-8vqf"
server_key: "26a10dd3c97c16d3d29ced92d7c81402"
server_key_iv: "8d9108f039bbbdeffca955d48f669d5b"
server_key_version: "encryptionkey@0"
I20260927 07:11:37.290740 18941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "387bbe581a214b7fbc242efa489ea382"
format_stamp: "Formatted at 2026-09-27 07:11:37 on dist-test-slave-8vqf"
server_key: "26a10dd3c97c16d3d29ced92d7c81402"
server_key_iv: "8d9108f039bbbdeffca955d48f669d5b"
server_key_version: "encryptionkey@0"
I20260927 07:11:37.296043 18941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:11:37.300021 18956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:37.301501 18941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:37.301693 18941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "387bbe581a214b7fbc242efa489ea382"
format_stamp: "Formatted at 2026-09-27 07:11:37 on dist-test-slave-8vqf"
server_key: "26a10dd3c97c16d3d29ced92d7c81402"
server_key_iv: "8d9108f039bbbdeffca955d48f669d5b"
server_key_version: "encryptionkey@0"
I20260927 07:11:37.301928 18941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:37.308002 18941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:37.312808 18941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:37.313135 18941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:37.332724 18941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39423
I20260927 07:11:37.332768 19007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39423 every 8 connection(s)
I20260927 07:11:37.334216 18941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:37.338171 19008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:37.338157 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 18941
I20260927 07:11:37.338370 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:37.338933 30785 external_mini_cluster.cc:1428] Setting key 0c8b27f9e3563cf9f8b6c7b8fde23e28
I20260927 07:11:37.347095 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:37.340676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38417 (local address 127.0.0.1:39423)
0927 07:11:37.341564 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:37.341576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:37.341618 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:11:37.343283 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:37.343334 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:37.343736 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:37.344187 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:37.345359 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:37.345365 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:37.345379 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:37.345392 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:37.345450 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:37.345472 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:37.345740 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:37.345885 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 07:11:37.346076 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":325,"threads_started":1}
I20260927 07:11:37.350967 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:37.352089 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382: Bootstrap starting.
I20260927 07:11:37.355614 19008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:37.357105 19008 log.cc:824] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:37.358176 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:37.355627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60859 (local address 127.0.0.1:39423)
0927 07:11:37.355832 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:37.355837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:37.356149 (+ 312us) server_negotiation.cc:407] Connection header received
0927 07:11:37.356358 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:37.356361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:37.356414 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:37.356490 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:37.357442 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:37.357446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:37.357449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:37.357463 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:37.357502 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:37.357517 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:37.357599 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:37.357794 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 07:11:37.357869 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 07:11:37.360342 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382: No bootstrap required, opened a new log
I20260927 07:11:37.367985 19008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } }
I20260927 07:11:37.368428 19008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:37.368561 19008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387bbe581a214b7fbc242efa489ea382, State: Initialized, Role: FOLLOWER
I20260927 07:11:37.369234 19008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } }
I20260927 07:11:37.369454 19008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:37.369551 19008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:37.369673 19008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:37.371984 19008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } }
I20260927 07:11:37.372465 19008 leader_election.cc:302] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 387bbe581a214b7fbc242efa489ea382; no voters:
I20260927 07:11:37.373049 19008 leader_election.cc:288] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:37.373220 19018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:37.373914 19018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [term 1 LEADER]: Becoming Leader. State: Replica: 387bbe581a214b7fbc242efa489ea382, State: Running, Role: LEADER
I20260927 07:11:37.374509 19018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } }
I20260927 07:11:37.375401 19008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:37.377679 19019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "387bbe581a214b7fbc242efa489ea382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } } }
I20260927 07:11:37.377643 19020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 387bbe581a214b7fbc242efa489ea382. Latest consensus state: current_term: 1 leader_uuid: "387bbe581a214b7fbc242efa489ea382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387bbe581a214b7fbc242efa489ea382" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39423 } } }
I20260927 07:11:37.378197 19019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:37.378198 19020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:37.380924 19025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:37.386893 19025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:37.397339 19025 catalog_manager.cc:1347] Generated new cluster ID: 1a476f6309574a499700df5061267724
I20260927 07:11:37.397482 19025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:37.424754 19025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:37.425235 19025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:37.439653 19025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 387bbe581a214b7fbc242efa489ea382: Generated new TSK 0
I20260927 07:11:37.440783 19025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:37.459635 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18941
W20260927 07:11:37.460813 19003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38417
W20260927 07:11:37.461490 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 18941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:37.461640 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 18941
2026-09-27T07:11:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:37 dist-test-slave-8vqf krb5kdc[19039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:37 dist-test-slave-8vqf krb5kdc[19039](info): set up 2 sockets
Sep 27 07:11:37 dist-test-slave-8vqf krb5kdc[19039](info): commencing operation
krb5kdc: starting...
W20260927 07:11:38.579047 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:38 dist-test-slave-8vqf krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:38.841835 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:39.178488 19052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:39.178831 19052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:39.178928 19052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:39.186275 19052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:39.186411 19052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:39.186460 19052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:39.186515 19052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:39.186583 19052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:39.197446 19052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:39.198858 19052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:39.200206 19052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:39.212605 19057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:39.212636 19058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:39.212674 19060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:39.213727 19052 server_base.cc:1034] running on GCE node
I20260927 07:11:39.214718 19052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:39.216270 19052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:39.217521 19052 hybrid_clock.cc:648] HybridClock initialized: now 1790493099217464 us; error 53 us; skew 500 ppm
I20260927 07:11:39.217931 19052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:39 dist-test-slave-8vqf krb5kdc[19039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:39.221529 19052 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:39.223244 19052 webserver.cc:460] Webserver started at http://127.0.0.1:36989/ using document root <none> and password file <none>
I20260927 07:11:39.223980 19052 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:39.224213 19052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:39.224538 19052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:39.227695 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "001e8f22e1d4410e953b86d1af73bf9a"
format_stamp: "Formatted at 2026-09-27 07:11:39 on dist-test-slave-8vqf"
server_key: "e14344da3b2de564db82fd557e9e6b78"
server_key_iv: "a0452d4d26b908bdddd3b6aac8882596"
server_key_version: "encryptionkey@0"
I20260927 07:11:39.228466 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "001e8f22e1d4410e953b86d1af73bf9a"
format_stamp: "Formatted at 2026-09-27 07:11:39 on dist-test-slave-8vqf"
server_key: "e14344da3b2de564db82fd557e9e6b78"
server_key_iv: "a0452d4d26b908bdddd3b6aac8882596"
server_key_version: "encryptionkey@0"
I20260927 07:11:39.233758 19052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:11:39.237231 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:39.238694 19052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:39.238885 19052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "001e8f22e1d4410e953b86d1af73bf9a"
format_stamp: "Formatted at 2026-09-27 07:11:39 on dist-test-slave-8vqf"
server_key: "e14344da3b2de564db82fd557e9e6b78"
server_key_iv: "a0452d4d26b908bdddd3b6aac8882596"
server_key_version: "encryptionkey@0"
I20260927 07:11:39.239082 19052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:39.264029 19052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:39.268620 19052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:39.268936 19052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:39.287571 19052 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41067
I20260927 07:11:39.287604 19118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41067 every 8 connection(s)
I20260927 07:11:39.288988 19052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:39.292908 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:39.296996 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19052
I20260927 07:11:39.297196 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:39.297724 30785 external_mini_cluster.cc:1428] Setting key cb696ef01107cf4ef1a8d77f54b44152
I20260927 07:11:39.305542 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a: Bootstrap starting.
I20260927 07:11:39.308770 19119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:39.309818 19119 log.cc:824] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:11:39 dist-test-slave-8vqf krb5kdc[19039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:11:39.313009 19119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a: No bootstrap required, opened a new log
I20260927 07:11:39.321327 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:39.299645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49703 (local address 127.0.0.1:41067)
0927 07:11:39.300398 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:39.300411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:39.300453 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:11:39.302096 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:39.302119 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:39.302617 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:39.303063 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:39.304574 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:39.305935 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:39.307292 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:39.307963 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:39.308153 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:39.312403 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:39.312414 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:39.312433 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:39.312482 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:39.316567 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:39.317288 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:39.317297 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:39.317307 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:39.317519 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:39.317899 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:39.317904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:39.317907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:39.318676 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:39.319011 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:39.319355 (+ 344us) server_negotiation.cc:299] Negotiation successful
0927 07:11:39.319857 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":329,"threads_started":1}
I20260927 07:11:39.324054 19119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } }
I20260927 07:11:39.324589 19119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:39.324733 19119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001e8f22e1d4410e953b86d1af73bf9a, State: Initialized, Role: FOLLOWER
I20260927 07:11:39.325525 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:39.325433 19119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } }
I20260927 07:11:39.325680 19119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:39.325827 19119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:39.326016 19119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:39.328342 19119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } }
I20260927 07:11:39.328953 19119 leader_election.cc:302] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 001e8f22e1d4410e953b86d1af73bf9a; no voters:
I20260927 07:11:39.329732 19119 leader_election.cc:288] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:39.329841 19128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:39.332494 19119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:39.335706 19128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [term 1 LEADER]: Becoming Leader. State: Replica: 001e8f22e1d4410e953b86d1af73bf9a, State: Running, Role: LEADER
I20260927 07:11:39.336558 19128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } }
I20260927 07:11:39.340031 19129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "001e8f22e1d4410e953b86d1af73bf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } } }
I20260927 07:11:39.340513 19129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:39.353168 19130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 001e8f22e1d4410e953b86d1af73bf9a. Latest consensus state: current_term: 1 leader_uuid: "001e8f22e1d4410e953b86d1af73bf9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "001e8f22e1d4410e953b86d1af73bf9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41067 } } }
I20260927 07:11:39.353876 19130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:39.355281 19136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:39.357081 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:39.330783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55303 (local address 127.0.0.1:41067)
0927 07:11:39.330956 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:39.330962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:39.335232 (+ 4270us) server_negotiation.cc:407] Connection header received
0927 07:11:39.335491 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:39.335500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:39.335604 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:39.335733 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:39.337354 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:39.338174 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:39.340010 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:39.340706 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:39.340889 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:39.345624 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:39.345635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:39.345643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:39.345703 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:39.350720 (+ 5017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:39.351982 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:39.351991 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:39.351997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:39.353643 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:39.354339 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:39.354346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:39.354349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:39.355048 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:39.355363 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:39.356175 (+ 812us) server_negotiation.cc:299] Negotiation successful
0927 07:11:39.356536 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:11:39.363677 19136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:39.374073 19136 catalog_manager.cc:1347] Generated new cluster ID: d6645044332b44d1949c508ed6a84f46
I20260927 07:11:39.374214 19136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:39.385870 19136 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:39.387696 19136 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:39.396310 19136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 001e8f22e1d4410e953b86d1af73bf9a: Generated new TSK 0
I20260927 07:11:39.397236 19136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:39.405906 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19052
I20260927 07:11:39.407269 19114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:39.501410 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19052
2026-09-27T07:11:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2040 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:39 dist-test-slave-8vqf krb5kdc[19152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:39 dist-test-slave-8vqf krb5kdc[19152](info): set up 2 sockets
Sep 27 07:11:39 dist-test-slave-8vqf krb5kdc[19152](info): commencing operation
krb5kdc: starting...
W20260927 07:11:40.619145 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:40 dist-test-slave-8vqf krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:40.879607 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:41.231448 19165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:41.231793 19165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:41.231994 19165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:41.239704 19165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:41.239847 19165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:41.239977 19165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:41.240095 19165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:41.240224 19165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:41.250643 19165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:41.252110 19165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:41.253561 19165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:41.266659 19170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:41.266690 19173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:41.266714 19171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:41.267632 19165 server_base.cc:1034] running on GCE node
I20260927 07:11:41.268419 19165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:41.269876 19165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:41.271073 19165 hybrid_clock.cc:648] HybridClock initialized: now 1790493101271006 us; error 57 us; skew 500 ppm
I20260927 07:11:41.271435 19165 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:41 dist-test-slave-8vqf krb5kdc[19152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:41.281320 19165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:41.283118 19165 webserver.cc:460] Webserver started at http://127.0.0.1:38007/ using document root <none> and password file <none>
I20260927 07:11:41.283867 19165 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:41.284008 19165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:41.284339 19165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:41.287950 19165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "14f3314602e74fa2a73d4ee8f3c01560"
format_stamp: "Formatted at 2026-09-27 07:11:41 on dist-test-slave-8vqf"
server_key: "a82d3db9ab8504badbd844537ca77d5c"
server_key_iv: "632fc7d74758df01800c2f4e41bdfb79"
server_key_version: "encryptionkey@0"
I20260927 07:11:41.288932 19165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "14f3314602e74fa2a73d4ee8f3c01560"
format_stamp: "Formatted at 2026-09-27 07:11:41 on dist-test-slave-8vqf"
server_key: "a82d3db9ab8504badbd844537ca77d5c"
server_key_iv: "632fc7d74758df01800c2f4e41bdfb79"
server_key_version: "encryptionkey@0"
I20260927 07:11:41.294826 19165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:11:41.298595 19180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:41.300010 19165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:41.300205 19165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "14f3314602e74fa2a73d4ee8f3c01560"
format_stamp: "Formatted at 2026-09-27 07:11:41 on dist-test-slave-8vqf"
server_key: "a82d3db9ab8504badbd844537ca77d5c"
server_key_iv: "632fc7d74758df01800c2f4e41bdfb79"
server_key_version: "encryptionkey@0"
I20260927 07:11:41.300436 19165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:41.330951 19165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:41.335206 19165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:41.335521 19165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:41.355170 19165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43481
I20260927 07:11:41.355189 19231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43481 every 8 connection(s)
I20260927 07:11:41.356762 19165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:41.360621 19232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:41.366470 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19165
I20260927 07:11:41.366683 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:41.367261 30785 external_mini_cluster.cc:1428] Setting key 8207179381af2e90f1f26e79568d5776
I20260927 07:11:41.372502 19232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560: Bootstrap starting.
I20260927 07:11:41.376438 19232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:41.377712 19232 log.cc:824] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:41.381044 19232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560: No bootstrap required, opened a new log
I20260927 07:11:41.381335 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:41.369229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47815 (local address 127.0.0.1:43481)
0927 07:11:41.370067 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:41.370080 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:41.370121 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:11:41.371887 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:41.371911 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:41.372345 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:41.372748 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:41.373314 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:41.374705 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:41.376632 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:41.377372 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:41.377504 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:41.378888 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:41.378901 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:41.378929 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:41.379620 (+ 691us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:41.379696 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:41.379721 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:41.379807 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:41.380029 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 07:11:41.380292 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":376,"threads_started":1}
I20260927 07:11:41.385109 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:41.391372 19232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } }
I20260927 07:11:41.391943 19232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:41.392073 19232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f3314602e74fa2a73d4ee8f3c01560, State: Initialized, Role: FOLLOWER
I20260927 07:11:41.392707 19232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } }
I20260927 07:11:41.392972 19232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:41.393131 19232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:41.393312 19232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:41.395705 19232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } }
I20260927 07:11:41.396250 19232 leader_election.cc:302] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f3314602e74fa2a73d4ee8f3c01560; no voters:
I20260927 07:11:41.397001 19232 leader_election.cc:288] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:41.397128 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:41.390537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48265 (local address 127.0.0.1:43481)
0927 07:11:41.390780 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:41.390786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:41.391093 (+ 307us) server_negotiation.cc:407] Connection header received
0927 07:11:41.391297 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:41.391301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:41.391385 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:41.391467 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:41.392086 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:41.393014 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:41.394498 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:41.395037 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:41.395175 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:41.396274 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:41.396278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:41.396281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:41.396298 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:41.396337 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:41.396353 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:41.396450 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:41.396612 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:11:41.396703 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 07:11:41.397131 19242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:41.399844 19232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:41.402256 19242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [term 1 LEADER]: Becoming Leader. State: Replica: 14f3314602e74fa2a73d4ee8f3c01560, State: Running, Role: LEADER
I20260927 07:11:41.402992 19242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } }
I20260927 07:11:41.406735 19244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14f3314602e74fa2a73d4ee8f3c01560. Latest consensus state: current_term: 1 leader_uuid: "14f3314602e74fa2a73d4ee8f3c01560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } } }
I20260927 07:11:41.406769 19243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14f3314602e74fa2a73d4ee8f3c01560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f3314602e74fa2a73d4ee8f3c01560" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43481 } } }
I20260927 07:11:41.407461 19244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:41.407495 19243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:41.408595 19252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:41.419023 19252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:41.430476 19252 catalog_manager.cc:1347] Generated new cluster ID: f309210b0a984735a483f2acfad84fd1
I20260927 07:11:41.430676 19252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:41.459033 19252 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:41.460636 19252 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:41.479135 19252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14f3314602e74fa2a73d4ee8f3c01560: Generated new TSK 0
I20260927 07:11:41.480202 19252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:41.499876 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19165
W20260927 07:11:41.501128 19227 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47815
W20260927 07:11:41.501962 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19165: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:41.502121 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19165
2026-09-27T07:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2000 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:41 dist-test-slave-8vqf krb5kdc[19263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:41 dist-test-slave-8vqf krb5kdc[19263](info): set up 2 sockets
Sep 27 07:11:41 dist-test-slave-8vqf krb5kdc[19263](info): commencing operation
krb5kdc: starting...
W20260927 07:11:42.627208 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:42 dist-test-slave-8vqf krb5kdc[19263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:42.892128 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:43.232412 19276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:43.232767 19276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:43.232934 19276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:43.240476 19276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:43.240645 19276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:43.240756 19276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:43.240866 19276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:43.240954 19276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:43.251060 19276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:43.252470 19276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:43.253857 19276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:43.266343 19281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:43.266364 19282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:43.266526 19284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:43.267190 19276 server_base.cc:1034] running on GCE node
I20260927 07:11:43.268162 19276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:43.269618 19276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:43.270852 19276 hybrid_clock.cc:648] HybridClock initialized: now 1790493103270811 us; error 63 us; skew 500 ppm
I20260927 07:11:43.271209 19276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:43 dist-test-slave-8vqf krb5kdc[19263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:43.274401 19276 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:43.275942 19276 webserver.cc:460] Webserver started at http://127.0.0.1:46005/ using document root <none> and password file <none>
I20260927 07:11:43.276737 19276 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:43.276880 19276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:43.277190 19276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:43.280288 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "df5fc7bade944433bc8f897a26320488"
format_stamp: "Formatted at 2026-09-27 07:11:43 on dist-test-slave-8vqf"
server_key: "34b700b30161e4c63218097fc1a574d4"
server_key_iv: "1cf9f1c7ba571dada0a432f181f3bda0"
server_key_version: "encryptionkey@0"
I20260927 07:11:43.281102 19276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "df5fc7bade944433bc8f897a26320488"
format_stamp: "Formatted at 2026-09-27 07:11:43 on dist-test-slave-8vqf"
server_key: "34b700b30161e4c63218097fc1a574d4"
server_key_iv: "1cf9f1c7ba571dada0a432f181f3bda0"
server_key_version: "encryptionkey@0"
I20260927 07:11:43.286451 19276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:11:43.290063 19291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:43.291495 19276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:11:43.291646 19276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "df5fc7bade944433bc8f897a26320488"
format_stamp: "Formatted at 2026-09-27 07:11:43 on dist-test-slave-8vqf"
server_key: "34b700b30161e4c63218097fc1a574d4"
server_key_iv: "1cf9f1c7ba571dada0a432f181f3bda0"
server_key_version: "encryptionkey@0"
I20260927 07:11:43.291816 19276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:43.297498 19276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:43.301955 19276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:43.302204 19276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:43.320408 19276 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45737
I20260927 07:11:43.320411 19342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45737 every 8 connection(s)
I20260927 07:11:43.321882 19276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:43.326114 19343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:43.327311 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19276
I20260927 07:11:43.327489 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:43.327941 30785 external_mini_cluster.cc:1428] Setting key 1e9d2a992b4bceec18322355eb8f5efe
I20260927 07:11:43.336131 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:43.329864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38565 (local address 127.0.0.1:45737)
0927 07:11:43.330774 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:43.330789 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:43.330841 (+ 52us) server_negotiation.cc:407] Connection header received
0927 07:11:43.332548 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:43.332577 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:43.332948 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:43.333361 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:43.334371 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:43.334378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:43.334390 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:43.334405 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:43.334467 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:43.334486 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:43.334739 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:43.334884 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 07:11:43.335070 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":380,"threads_started":1}
I20260927 07:11:43.339751 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488: Bootstrap starting.
I20260927 07:11:43.340024 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:43.343954 19343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:43.345364 19343 log.cc:824] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:43.348183 19346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:43.345476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38151 (local address 127.0.0.1:45737)
0927 07:11:43.345629 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:43.345633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:43.346069 (+ 436us) server_negotiation.cc:407] Connection header received
0927 07:11:43.346258 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:43.346264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:43.346330 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:43.346476 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:43.347495 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:43.347500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:43.347503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:43.347518 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:43.347555 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:43.347570 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:43.347671 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:43.347826 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 07:11:43.347881 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:11:43.348877 19343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488: No bootstrap required, opened a new log
I20260927 07:11:43.357347 19343 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260927 07:11:43.357776 19343 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:43.357887 19343 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5fc7bade944433bc8f897a26320488, State: Initialized, Role: FOLLOWER
I20260927 07:11:43.358541 19343 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260927 07:11:43.358858 19343 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:43.359006 19343 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:43.359193 19343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:43.361668 19343 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260927 07:11:43.362244 19343 leader_election.cc:302] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5fc7bade944433bc8f897a26320488; no voters:
I20260927 07:11:43.362980 19343 leader_election.cc:288] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:43.363112 19353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:43.363808 19353 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [term 1 LEADER]: Becoming Leader. State: Replica: df5fc7bade944433bc8f897a26320488, State: Running, Role: LEADER
I20260927 07:11:43.364322 19353 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } }
I20260927 07:11:43.365123 19343 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:43.367403 19355 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5fc7bade944433bc8f897a26320488. Latest consensus state: current_term: 1 leader_uuid: "df5fc7bade944433bc8f897a26320488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260927 07:11:43.367940 19355 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:43.370468 19362 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:43.370393 19354 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df5fc7bade944433bc8f897a26320488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fc7bade944433bc8f897a26320488" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45737 } } }
I20260927 07:11:43.370776 19354 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:43.376134 19362 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:43.386020 19362 catalog_manager.cc:1347] Generated new cluster ID: 1466e871c8d74598b9ad55781bd25b6e
I20260927 07:11:43.386152 19362 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:43.394912 19362 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:43.395445 19362 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:43.405889 19362 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df5fc7bade944433bc8f897a26320488: Generated new TSK 0
I20260927 07:11:43.406841 19362 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:43.415133 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19276
W20260927 07:11:43.416251 19338 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38565
W20260927 07:11:43.417024 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:43.417192 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19276
2026-09-27T07:11:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:43 dist-test-slave-8vqf krb5kdc[19374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:43 dist-test-slave-8vqf krb5kdc[19374](info): set up 2 sockets
Sep 27 07:11:43 dist-test-slave-8vqf krb5kdc[19374](info): commencing operation
krb5kdc: starting...
W20260927 07:11:44.527127 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:44 dist-test-slave-8vqf krb5kdc[19374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:44.792178 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:45.145865 19387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:45.146304 19387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:45.146489 19387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:45.154825 19387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:45.155010 19387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:45.155118 19387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:45.155238 19387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:45.155318 19387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:45.166029 19387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:45.167995 19387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:45.169706 19387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:45.183934 19395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:45.184059 19393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:45.184105 19392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:45.185267 19387 server_base.cc:1034] running on GCE node
I20260927 07:11:45.186146 19387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:45.187781 19387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:45.189023 19387 hybrid_clock.cc:648] HybridClock initialized: now 1790493105188958 us; error 62 us; skew 500 ppm
I20260927 07:11:45.189401 19387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:45 dist-test-slave-8vqf krb5kdc[19374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:45.192643 19387 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:45.194175 19387 webserver.cc:460] Webserver started at http://127.0.0.1:38959/ using document root <none> and password file <none>
I20260927 07:11:45.194926 19387 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:45.195032 19387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:45.195282 19387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:45.198288 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "26e95c0d5fe548e38762d09dafd88755"
format_stamp: "Formatted at 2026-09-27 07:11:45 on dist-test-slave-8vqf"
server_key: "e95386c9ecdb22e002dca520c0d61e94"
server_key_iv: "e169afc07a6b8dfe93610007c12d0228"
server_key_version: "encryptionkey@0"
I20260927 07:11:45.199120 19387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "26e95c0d5fe548e38762d09dafd88755"
format_stamp: "Formatted at 2026-09-27 07:11:45 on dist-test-slave-8vqf"
server_key: "e95386c9ecdb22e002dca520c0d61e94"
server_key_iv: "e169afc07a6b8dfe93610007c12d0228"
server_key_version: "encryptionkey@0"
I20260927 07:11:45.204631 19387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:45.208101 19402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:45.209496 19387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:45.209642 19387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "26e95c0d5fe548e38762d09dafd88755"
format_stamp: "Formatted at 2026-09-27 07:11:45 on dist-test-slave-8vqf"
server_key: "e95386c9ecdb22e002dca520c0d61e94"
server_key_iv: "e169afc07a6b8dfe93610007c12d0228"
server_key_version: "encryptionkey@0"
I20260927 07:11:45.209815 19387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:45.229245 19387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:45.233381 19387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:45.233624 19387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:45.253363 19387 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46207
I20260927 07:11:45.253454 19453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46207 every 8 connection(s)
I20260927 07:11:45.254949 19387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:45.258755 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19387
I20260927 07:11:45.258947 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:45.259161 19454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:45.259493 30785 external_mini_cluster.cc:1428] Setting key c379ace3c6f108ca28f68f0aeafc34be
I20260927 07:11:45.274360 19454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755: Bootstrap starting.
Sep 27 07:11:45 dist-test-slave-8vqf krb5kdc[19374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:11:45.278647 19454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:45.280088 19454 log.cc:824] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:45.283685 19454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755: No bootstrap required, opened a new log
I20260927 07:11:45.285049 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:45.261370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:53191 (local address 127.0.0.1:46207)
0927 07:11:45.262226 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:45.262247 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:45.262304 (+ 57us) server_negotiation.cc:407] Connection header received
0927 07:11:45.264309 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:45.264342 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:45.264934 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:45.265448 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:45.266786 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:45.268794 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:45.270161 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:45.271325 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:45.271540 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:45.275847 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:45.275859 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:45.275883 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:45.275942 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:45.280051 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:45.281133 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:45.281140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:45.281147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:45.281323 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:45.281799 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:45.281803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:45.281806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:45.282499 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:45.282859 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:45.283266 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 07:11:45.283610 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":356,"threads_started":1}
I20260927 07:11:45.289284 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:45.292888 19454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260927 07:11:45.293303 19454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:45.293464 19454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e95c0d5fe548e38762d09dafd88755, State: Initialized, Role: FOLLOWER
I20260927 07:11:45.294320 19454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260927 07:11:45.294672 19454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:45.294833 19454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:45.295046 19454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:45.298063 19454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260927 07:11:45.298712 19454 leader_election.cc:302] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26e95c0d5fe548e38762d09dafd88755; no voters:
I20260927 07:11:45.299472 19454 leader_election.cc:288] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:45.299635 19464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:45.301822 19454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:45.306671 19464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [term 1 LEADER]: Becoming Leader. State: Replica: 26e95c0d5fe548e38762d09dafd88755, State: Running, Role: LEADER
I20260927 07:11:45.307374 19464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } }
I20260927 07:11:45.311599 19465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26e95c0d5fe548e38762d09dafd88755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } } }
I20260927 07:11:45.312247 19465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:45.311628 19466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26e95c0d5fe548e38762d09dafd88755. Latest consensus state: current_term: 1 leader_uuid: "26e95c0d5fe548e38762d09dafd88755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26e95c0d5fe548e38762d09dafd88755" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46207 } } }
I20260927 07:11:45.312729 19466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:45.313791 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:45.293566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:58001 (local address 127.0.0.1:46207)
0927 07:11:45.293795 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:45.293804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:45.294216 (+ 412us) server_negotiation.cc:407] Connection header received
0927 07:11:45.294407 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:45.294410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:45.294473 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:45.294603 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:45.296567 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:45.297323 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:45.299050 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:45.299959 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:45.300128 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:45.304526 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:45.304536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:45.304543 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:45.304604 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:45.309178 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:45.310095 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:45.310099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:45.310101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:45.310220 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:45.310809 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:45.310815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:45.310818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:45.311302 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:45.312819 (+ 1517us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:45.313028 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 07:11:45.313267 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:11:45.317071 19479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:45.322502 19479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:45.331670 19479 catalog_manager.cc:1347] Generated new cluster ID: ec64c7e757e64ef084abda473e3cdb9f
I20260927 07:11:45.331804 19479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:45.348444 19479 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:45.349830 19479 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:45.358135 19479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26e95c0d5fe548e38762d09dafd88755: Generated new TSK 0
I20260927 07:11:45.359297 19479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:45.375031 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19387
I20260927 07:11:45.376340 19449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:45.473201 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19387
W20260927 07:11:45.490891 19372 connection.cc:535] client connection to 127.0.0.1:46207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46207): Connection reset by peer (error 104)
2026-09-27T07:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2056 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:45 dist-test-slave-8vqf krb5kdc[19486](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:45 dist-test-slave-8vqf krb5kdc[19486](info): set up 2 sockets
Sep 27 07:11:45 dist-test-slave-8vqf krb5kdc[19486](info): commencing operation
krb5kdc: starting...
W20260927 07:11:46.619125 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:46 dist-test-slave-8vqf krb5kdc[19486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:46.884102 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:47.227164 19499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:47.227483 19499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:47.227653 19499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:47.235101 19499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:47.235250 19499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:47.235419 19499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:47.235518 19499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:47.235600 19499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:47.246227 19499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:47.247643 19499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:47.249013 19499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:47.261562 19504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:47.261479 19507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:47.261511 19505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:47.262459 19499 server_base.cc:1034] running on GCE node
I20260927 07:11:47.263368 19499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:47.264885 19499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:47.266089 19499 hybrid_clock.cc:648] HybridClock initialized: now 1790493107266039 us; error 39 us; skew 500 ppm
I20260927 07:11:47.266445 19499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:47 dist-test-slave-8vqf krb5kdc[19486](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:47.269562 19499 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:47.271205 19499 webserver.cc:460] Webserver started at http://127.0.0.1:45717/ using document root <none> and password file <none>
I20260927 07:11:47.271912 19499 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:47.272065 19499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:47.272377 19499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:47.275959 19499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "4a9fca6fa7c5417c84c646b848fab090"
format_stamp: "Formatted at 2026-09-27 07:11:47 on dist-test-slave-8vqf"
server_key: "9ddbbc9e7cb8702499248e2faf6cedba"
server_key_iv: "5cc691e9f85d9886c9d3dffb648571a1"
server_key_version: "encryptionkey@0"
I20260927 07:11:47.276793 19499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "4a9fca6fa7c5417c84c646b848fab090"
format_stamp: "Formatted at 2026-09-27 07:11:47 on dist-test-slave-8vqf"
server_key: "9ddbbc9e7cb8702499248e2faf6cedba"
server_key_iv: "5cc691e9f85d9886c9d3dffb648571a1"
server_key_version: "encryptionkey@0"
I20260927 07:11:47.282313 19499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:11:47.285931 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:47.287348 19499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:11:47.287532 19499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "4a9fca6fa7c5417c84c646b848fab090"
format_stamp: "Formatted at 2026-09-27 07:11:47 on dist-test-slave-8vqf"
server_key: "9ddbbc9e7cb8702499248e2faf6cedba"
server_key_iv: "5cc691e9f85d9886c9d3dffb648571a1"
server_key_version: "encryptionkey@0"
I20260927 07:11:47.287740 19499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:47.311288 19499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:47.315783 19499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:47.316110 19499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:47.336257 19499 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34189
I20260927 07:11:47.336302 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34189 every 8 connection(s)
I20260927 07:11:47.337767 19499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:47.340837 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19499
I20260927 07:11:47.341028 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:47.341578 30785 external_mini_cluster.cc:1428] Setting key b7f196b456925a0eb30ea4058546c790
I20260927 07:11:47.342420 19566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:47.355341 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:47.343605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:39899 (local address 127.0.0.1:34189)
0927 07:11:47.344485 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:47.344497 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:47.344538 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:11:47.346135 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:47.346158 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:47.346673 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:47.347106 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:47.347681 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:47.349055 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:47.350451 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:47.351218 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:47.351430 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:47.352799 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:47.352808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:47.352827 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:47.353459 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:47.353542 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:47.353575 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:47.353765 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:47.354050 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 07:11:47.354238 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":372,"threads_started":1}
I20260927 07:11:47.355809 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090: Bootstrap starting.
I20260927 07:11:47.359485 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:47.359527 19566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:47.360553 19566 log.cc:824] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:47.363606 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090: No bootstrap required, opened a new log
I20260927 07:11:47.370875 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:47.363998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:50567 (local address 127.0.0.1:34189)
0927 07:11:47.364219 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:47.364224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:47.364604 (+ 380us) server_negotiation.cc:407] Connection header received
0927 07:11:47.364768 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:47.364773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:47.364852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:47.364981 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:47.365456 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:47.366684 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:47.368001 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:47.368733 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:47.368879 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:47.370013 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:47.370020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:47.370025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:47.370050 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:47.370108 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:47.370130 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:47.370257 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:47.370427 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 07:11:47.370486 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:11:47.373605 19566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260927 07:11:47.374054 19566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:47.374188 19566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9fca6fa7c5417c84c646b848fab090, State: Initialized, Role: FOLLOWER
I20260927 07:11:47.374905 19566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260927 07:11:47.375154 19566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:47.375310 19566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:47.375494 19566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:47.377861 19566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260927 07:11:47.378398 19566 leader_election.cc:302] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a9fca6fa7c5417c84c646b848fab090; no voters:
I20260927 07:11:47.379038 19566 leader_election.cc:288] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:47.379237 19576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:47.380604 19576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [term 1 LEADER]: Becoming Leader. State: Replica: 4a9fca6fa7c5417c84c646b848fab090, State: Running, Role: LEADER
I20260927 07:11:47.381228 19576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260927 07:11:47.381337 19566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:47.384634 19577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a9fca6fa7c5417c84c646b848fab090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260927 07:11:47.385304 19577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:47.384910 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a9fca6fa7c5417c84c646b848fab090. Latest consensus state: current_term: 1 leader_uuid: "4a9fca6fa7c5417c84c646b848fab090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a9fca6fa7c5417c84c646b848fab090" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260927 07:11:47.385967 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:47.387147 19585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:47.393081 19585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:47.403087 19585 catalog_manager.cc:1347] Generated new cluster ID: 8843339eef1c44e1812bb23334bca790
I20260927 07:11:47.403226 19585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:47.417225 19585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:47.418689 19585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:47.433174 19585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a9fca6fa7c5417c84c646b848fab090: Generated new TSK 0
I20260927 07:11:47.434232 19585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:47.444664 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19499
W20260927 07:11:47.446719 19561 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:39899
W20260927 07:11:47.447633 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19499: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:47.447803 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19499
W20260927 07:11:47.466184 19484 connection.cc:535] client connection to 127.0.0.1:34189 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34189): Connection reset by peer (error 104)
2026-09-27T07:11:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:47 dist-test-slave-8vqf krb5kdc[19597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:47 dist-test-slave-8vqf krb5kdc[19597](info): set up 2 sockets
Sep 27 07:11:47 dist-test-slave-8vqf krb5kdc[19597](info): commencing operation
krb5kdc: starting...
W20260927 07:11:48.567111 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:48 dist-test-slave-8vqf krb5kdc[19597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:48.833885 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:49.172542 19610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:49.172854 19610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:49.173030 19610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:49.180408 19610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:49.180536 19610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:49.180653 19610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:49.180761 19610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:49.180845 19610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:49.191300 19610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:49.192879 19610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:49.194228 19610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:49.206499 19618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:49.206588 19615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:49.206650 19616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:49.207365 19610 server_base.cc:1034] running on GCE node
I20260927 07:11:49.208182 19610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:49.209689 19610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:49.210924 19610 hybrid_clock.cc:648] HybridClock initialized: now 1790493109210873 us; error 48 us; skew 500 ppm
I20260927 07:11:49.211300 19610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:49 dist-test-slave-8vqf krb5kdc[19597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:49.214468 19610 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:49.216203 19610 webserver.cc:460] Webserver started at http://127.0.0.1:46681/ using document root <none> and password file <none>
I20260927 07:11:49.216912 19610 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:49.217043 19610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:49.217367 19610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:49.220516 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "de55395a66ec40cc8d59903008c1d7e3"
format_stamp: "Formatted at 2026-09-27 07:11:49 on dist-test-slave-8vqf"
server_key: "6474eefab2d5271db5e3a8ce829df1d6"
server_key_iv: "d1afa3d439eaacdee172c26265bd7bfc"
server_key_version: "encryptionkey@0"
I20260927 07:11:49.221298 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "de55395a66ec40cc8d59903008c1d7e3"
format_stamp: "Formatted at 2026-09-27 07:11:49 on dist-test-slave-8vqf"
server_key: "6474eefab2d5271db5e3a8ce829df1d6"
server_key_iv: "d1afa3d439eaacdee172c26265bd7bfc"
server_key_version: "encryptionkey@0"
I20260927 07:11:49.227273 19610 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 07:11:49.230800 19625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:49.232178 19610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:11:49.232373 19610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "de55395a66ec40cc8d59903008c1d7e3"
format_stamp: "Formatted at 2026-09-27 07:11:49 on dist-test-slave-8vqf"
server_key: "6474eefab2d5271db5e3a8ce829df1d6"
server_key_iv: "d1afa3d439eaacdee172c26265bd7bfc"
server_key_version: "encryptionkey@0"
I20260927 07:11:49.232575 19610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:49.238366 19610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:49.242364 19610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:49.242708 19610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:49.261762 19610 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44621
I20260927 07:11:49.261870 19676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44621 every 8 connection(s)
I20260927 07:11:49.263331 19610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:49.267210 19677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:49.270378 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19610
I20260927 07:11:49.270592 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:49.271117 30785 external_mini_cluster.cc:1428] Setting key 4e5ec4d098ff0d379fc982e4a8b7dbfc
I20260927 07:11:49.279129 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:49.273047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:53795 (local address 127.0.0.1:44621)
0927 07:11:49.273819 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:49.273831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:49.273879 (+ 48us) server_negotiation.cc:407] Connection header received
0927 07:11:49.275625 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:49.275647 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:49.276009 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:49.276418 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:49.277498 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:49.277507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:49.277520 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:49.277533 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:49.277585 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:49.277605 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:49.277833 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:49.277975 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 07:11:49.278160 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":329,"threads_started":1}
I20260927 07:11:49.279608 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3: Bootstrap starting.
I20260927 07:11:49.283366 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:49.283601 19677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:49.284672 19677 log.cc:824] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:49.287830 19677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3: No bootstrap required, opened a new log
I20260927 07:11:49.290638 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:49.288112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:45277 (local address 127.0.0.1:44621)
0927 07:11:49.288318 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:49.288328 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:49.288627 (+ 299us) server_negotiation.cc:407] Connection header received
0927 07:11:49.288801 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:49.288807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:49.288860 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:49.288939 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:49.289919 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:49.289923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:49.289926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:49.289941 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:49.289977 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:49.289995 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:49.290086 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:49.290212 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 07:11:49.290260 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:11:49.296731 19677 raft_consensus.cc:348] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260927 07:11:49.297219 19677 raft_consensus.cc:374] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:49.297333 19677 raft_consensus.cc:729] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de55395a66ec40cc8d59903008c1d7e3, State: Initialized, Role: FOLLOWER
I20260927 07:11:49.297986 19677 consensus_queue.cc:260] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260927 07:11:49.298230 19677 raft_consensus.cc:388] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:49.298380 19677 raft_consensus.cc:482] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:49.298586 19677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:49.300865 19677 raft_consensus.cc:504] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260927 07:11:49.301414 19677 leader_election.cc:302] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de55395a66ec40cc8d59903008c1d7e3; no voters:
I20260927 07:11:49.302034 19677 leader_election.cc:288] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:49.302181 19687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:49.302969 19687 raft_consensus.cc:686] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [term 1 LEADER]: Becoming Leader. State: Replica: de55395a66ec40cc8d59903008c1d7e3, State: Running, Role: LEADER
I20260927 07:11:49.303495 19687 consensus_queue.cc:237] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } }
I20260927 07:11:49.304345 19677 sys_catalog.cc:564] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:49.306866 19689 sys_catalog.cc:455] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de55395a66ec40cc8d59903008c1d7e3. Latest consensus state: current_term: 1 leader_uuid: "de55395a66ec40cc8d59903008c1d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } } }
I20260927 07:11:49.306851 19688 sys_catalog.cc:455] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de55395a66ec40cc8d59903008c1d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de55395a66ec40cc8d59903008c1d7e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44621 } } }
I20260927 07:11:49.307272 19689 sys_catalog.cc:458] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:49.307301 19688 sys_catalog.cc:458] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:49.311581 19694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:49.317085 19694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:49.326678 19694 catalog_manager.cc:1347] Generated new cluster ID: 820685e7c6314befb28935e4de2a54ed
I20260927 07:11:49.326812 19694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:49.354360 19694 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:49.354984 19694 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:49.368351 19694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de55395a66ec40cc8d59903008c1d7e3: Generated new TSK 0
I20260927 07:11:49.369554 19694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:49.374277 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19610
W20260927 07:11:49.375710 19672 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:53795
W20260927 07:11:49.376509 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19610: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:49.376669 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19610
2026-09-27T07:11:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:49 dist-test-slave-8vqf krb5kdc[19708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:49 dist-test-slave-8vqf krb5kdc[19708](info): set up 2 sockets
Sep 27 07:11:49 dist-test-slave-8vqf krb5kdc[19708](info): commencing operation
krb5kdc: starting...
W20260927 07:11:50.491225 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:50 dist-test-slave-8vqf krb5kdc[19708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:50.766328 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43099
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:51.106606 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:51.106977 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:51.107149 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:51.114645 19721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:51.114809 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:51.114919 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:51.115031 19721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:51.115108 19721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:51.125309 19721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43099
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:51.126770 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:51.128160 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:51.140170 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:51.140205 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:51.140239 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:51.141057 19721 server_base.cc:1034] running on GCE node
I20260927 07:11:51.141906 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:51.143529 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:51.144755 19721 hybrid_clock.cc:648] HybridClock initialized: now 1790493111144709 us; error 41 us; skew 500 ppm
I20260927 07:11:51.145140 19721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:51 dist-test-slave-8vqf krb5kdc[19708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:51.148679 19721 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:51.150417 19721 webserver.cc:460] Webserver started at http://127.0.0.1:40343/ using document root <none> and password file <none>
I20260927 07:11:51.151186 19721 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:51.151338 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:51.151664 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:51.154932 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "084b308681b242ae92788002f94cd4d7"
format_stamp: "Formatted at 2026-09-27 07:11:51 on dist-test-slave-8vqf"
server_key: "f42a1a082208df472513d281bf486ace"
server_key_iv: "aa792ead325ae39f834f24c14bc817a4"
server_key_version: "encryptionkey@0"
I20260927 07:11:51.155733 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "084b308681b242ae92788002f94cd4d7"
format_stamp: "Formatted at 2026-09-27 07:11:51 on dist-test-slave-8vqf"
server_key: "f42a1a082208df472513d281bf486ace"
server_key_iv: "aa792ead325ae39f834f24c14bc817a4"
server_key_version: "encryptionkey@0"
I20260927 07:11:51.161307 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:51.165107 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:51.166616 19721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 07:11:51.166818 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "084b308681b242ae92788002f94cd4d7"
format_stamp: "Formatted at 2026-09-27 07:11:51 on dist-test-slave-8vqf"
server_key: "f42a1a082208df472513d281bf486ace"
server_key_iv: "aa792ead325ae39f834f24c14bc817a4"
server_key_version: "encryptionkey@0"
I20260927 07:11:51.167037 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:51.188333 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:51.192757 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:51.193070 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:51.212405 19721 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43099
I20260927 07:11:51.212428 19787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43099 every 8 connection(s)
I20260927 07:11:51.213934 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:51.217965 19788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:51.222314 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19721
I20260927 07:11:51.222493 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:51.223110 30785 external_mini_cluster.cc:1428] Setting key de0030220822f56d0f39f8ab956240e4
I20260927 07:11:51.231237 19788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7: Bootstrap starting.
I20260927 07:11:51.234691 19788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:51.235770 19788 log.cc:824] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7: Log is configured to *not* fsync() on all Append() calls
Sep 27 07:11:51 dist-test-slave-8vqf krb5kdc[19708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 07:11:51.239205 19788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7: No bootstrap required, opened a new log
I20260927 07:11:51.246369 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:51.225148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:38535 (local address 127.0.0.1:43099)
0927 07:11:51.225933 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:51.225946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:51.225987 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:11:51.227678 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:51.227701 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:51.228172 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:51.228603 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:51.230023 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:51.231484 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:51.232832 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:51.233519 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:51.233715 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:51.238334 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:51.238342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:51.238357 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:51.238405 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:51.241780 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:51.242779 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:51.242786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:51.242792 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:51.243016 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:51.243510 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:51.243513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:51.243515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:51.244117 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:51.244378 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:51.244902 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 07:11:51.245254 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260927 07:11:51.248358 19788 raft_consensus.cc:348] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } }
I20260927 07:11:51.248809 19788 raft_consensus.cc:374] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:51.248967 19788 raft_consensus.cc:729] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 084b308681b242ae92788002f94cd4d7, State: Initialized, Role: FOLLOWER
I20260927 07:11:51.249627 19788 consensus_queue.cc:260] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } }
I20260927 07:11:51.249884 19788 raft_consensus.cc:388] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:51.250042 19788 raft_consensus.cc:482] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:51.250221 19788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:51.250680 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:51.252664 19788 raft_consensus.cc:504] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } }
I20260927 07:11:51.253265 19788 leader_election.cc:302] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 084b308681b242ae92788002f94cd4d7; no voters:
I20260927 07:11:51.253943 19788 leader_election.cc:288] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:51.254027 19795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:51.256489 19795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [term 1 LEADER]: Becoming Leader. State: Replica: 084b308681b242ae92788002f94cd4d7, State: Running, Role: LEADER
I20260927 07:11:51.256711 19788 sys_catalog.cc:564] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:51.257189 19795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } }
I20260927 07:11:51.267318 19798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "084b308681b242ae92788002f94cd4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } } }
I20260927 07:11:51.267392 19799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 084b308681b242ae92788002f94cd4d7. Latest consensus state: current_term: 1 leader_uuid: "084b308681b242ae92788002f94cd4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "084b308681b242ae92788002f94cd4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43099 } } }
I20260927 07:11:51.267825 19799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:51.267827 19798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:51.271523 19807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:51.274137 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:51.255560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:33937 (local address 127.0.0.1:43099)
0927 07:11:51.255798 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:51.255807 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:51.256412 (+ 605us) server_negotiation.cc:407] Connection header received
0927 07:11:51.256894 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:51.256899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:51.256984 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:51.257106 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:51.258979 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:51.260163 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:51.261699 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:51.262250 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:51.262403 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:51.265218 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:51.265226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:51.265235 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:51.265291 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:51.270354 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:51.271432 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:51.271438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:51.271444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:51.271628 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:51.272495 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:51.272501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:51.272503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:51.272815 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:51.273015 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:51.273481 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 07:11:51.273740 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:11:51.277099 19807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:51.285470 19807 catalog_manager.cc:1347] Generated new cluster ID: 7a09893a499740a4a9fac99de61c885c
I20260927 07:11:51.285598 19807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:51.298871 19807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:51.300196 19807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:51.312599 19807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 084b308681b242ae92788002f94cd4d7: Generated new TSK 0
I20260927 07:11:51.313759 19807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:51.328533 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19721
I20260927 07:11:51.329835 19783 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:51.423882 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19721
W20260927 07:11:51.441579 19706 connection.cc:535] client connection to 127.0.0.1:43099 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43099): Connection reset by peer (error 104)
2026-09-27T07:11:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2047 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:51 dist-test-slave-8vqf krb5kdc[19820](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:51 dist-test-slave-8vqf krb5kdc[19820](info): set up 2 sockets
Sep 27 07:11:51 dist-test-slave-8vqf krb5kdc[19820](info): commencing operation
krb5kdc: starting...
W20260927 07:11:52.555212 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:52 dist-test-slave-8vqf krb5kdc[19820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:52.831076 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:53.169554 19833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:53.169878 19833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:53.170045 19833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:53.177493 19833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:53.177655 19833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:53.177753 19833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:53.177868 19833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:53.177940 19833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:53.188272 19833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:53.189657 19833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:53.191052 19833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:53.204416 19838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:53.204427 19839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:53.204661 19841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:53.205181 19833 server_base.cc:1034] running on GCE node
I20260927 07:11:53.206110 19833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:53.207719 19833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:53.208974 19833 hybrid_clock.cc:648] HybridClock initialized: now 1790493113208918 us; error 69 us; skew 500 ppm
I20260927 07:11:53.209295 19833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:53 dist-test-slave-8vqf krb5kdc[19820](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:53.212488 19833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:53.213920 19833 webserver.cc:460] Webserver started at http://127.0.0.1:42321/ using document root <none> and password file <none>
I20260927 07:11:53.214680 19833 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:53.214787 19833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:53.215058 19833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:53.218190 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "828816c8c0f046c8b64dd114a80943a2"
format_stamp: "Formatted at 2026-09-27 07:11:53 on dist-test-slave-8vqf"
server_key: "81ce20cbdd6fb08dd11d6e1c21b21536"
server_key_iv: "3ca5fb12f3adcd507f11661c56d8a21e"
server_key_version: "encryptionkey@0"
I20260927 07:11:53.219019 19833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "828816c8c0f046c8b64dd114a80943a2"
format_stamp: "Formatted at 2026-09-27 07:11:53 on dist-test-slave-8vqf"
server_key: "81ce20cbdd6fb08dd11d6e1c21b21536"
server_key_iv: "3ca5fb12f3adcd507f11661c56d8a21e"
server_key_version: "encryptionkey@0"
I20260927 07:11:53.224438 19833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:11:53.228225 19848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:53.230028 19833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:53.230229 19833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "828816c8c0f046c8b64dd114a80943a2"
format_stamp: "Formatted at 2026-09-27 07:11:53 on dist-test-slave-8vqf"
server_key: "81ce20cbdd6fb08dd11d6e1c21b21536"
server_key_iv: "3ca5fb12f3adcd507f11661c56d8a21e"
server_key_version: "encryptionkey@0"
I20260927 07:11:53.230472 19833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:53.247309 19833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:53.251488 19833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:53.251797 19833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:53.270828 19833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36283
I20260927 07:11:53.270861 19899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36283 every 8 connection(s)
I20260927 07:11:53.272294 19833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:53.276193 19900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:53.277232 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19833
I20260927 07:11:53.277408 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:53.278023 30785 external_mini_cluster.cc:1428] Setting key abe40ae1f7459aa7fb3744360b983f1c
I20260927 07:11:53.289541 19900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2: Bootstrap starting.
I20260927 07:11:53.291975 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:53.279987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:42021 (local address 127.0.0.1:36283)
0927 07:11:53.281183 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:53.281199 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:53.281240 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:11:53.283210 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:53.283236 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:53.283685 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:53.284106 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:53.284657 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:53.285927 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:53.287319 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:53.288025 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:53.288224 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:53.289547 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:53.289555 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:53.289571 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:53.290191 (+ 620us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:53.290269 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:53.290295 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:53.290462 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:53.290801 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 07:11:53.291008 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":986,"thread_start_us":735,"threads_started":1}
I20260927 07:11:53.292995 19900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:53.294373 19900 log.cc:824] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:53.296260 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:53.297833 19900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2: No bootstrap required, opened a new log
I20260927 07:11:53.309799 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:53.302596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:56213 (local address 127.0.0.1:36283)
0927 07:11:53.302801 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:53.302807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:53.303261 (+ 454us) server_negotiation.cc:407] Connection header received
0927 07:11:53.303420 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:53.303428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:53.303529 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:53.303644 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:53.304212 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:53.305462 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:53.306949 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:53.307691 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:53.307822 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:53.308961 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:53.308966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:53.308971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:53.308993 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:53.309063 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:53.309089 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:53.309198 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:53.309372 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 07:11:53.309431 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 07:11:53.309610 19900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260927 07:11:53.310173 19900 raft_consensus.cc:374] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:53.310313 19900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 828816c8c0f046c8b64dd114a80943a2, State: Initialized, Role: FOLLOWER
I20260927 07:11:53.311133 19900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260927 07:11:53.311522 19900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:53.311689 19900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:53.311925 19900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:53.314428 19900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260927 07:11:53.315006 19900 leader_election.cc:302] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 828816c8c0f046c8b64dd114a80943a2; no voters:
I20260927 07:11:53.315615 19900 leader_election.cc:288] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:53.315761 19910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:53.316461 19910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [term 1 LEADER]: Becoming Leader. State: Replica: 828816c8c0f046c8b64dd114a80943a2, State: Running, Role: LEADER
I20260927 07:11:53.317104 19910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } }
I20260927 07:11:53.317880 19900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:53.320205 19911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "828816c8c0f046c8b64dd114a80943a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } } }
I20260927 07:11:53.320612 19911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:53.320344 19912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 828816c8c0f046c8b64dd114a80943a2. Latest consensus state: current_term: 1 leader_uuid: "828816c8c0f046c8b64dd114a80943a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "828816c8c0f046c8b64dd114a80943a2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36283 } } }
I20260927 07:11:53.320772 19912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:53.324115 19915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:53.329414 19915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:53.338393 19915 catalog_manager.cc:1347] Generated new cluster ID: 21abe31b85234671997b449590def1e9
I20260927 07:11:53.338546 19915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:53.353920 19915 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:53.355333 19915 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:53.363830 19915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 828816c8c0f046c8b64dd114a80943a2: Generated new TSK 0
I20260927 07:11:53.364830 19915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:53.373942 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19833
W20260927 07:11:53.375597 19895 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:42021
W20260927 07:11:53.376358 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19833: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:53.376533 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19833
W20260927 07:11:53.391151 19818 connection.cc:535] client connection to 127.0.0.1:36283 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36283): Connection reset by peer (error 104)
2026-09-27T07:11:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:53 dist-test-slave-8vqf krb5kdc[19931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:53 dist-test-slave-8vqf krb5kdc[19931](info): set up 2 sockets
Sep 27 07:11:53 dist-test-slave-8vqf krb5kdc[19931](info): commencing operation
krb5kdc: starting...
W20260927 07:11:54.495260 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:54 dist-test-slave-8vqf krb5kdc[19931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 07:11:54.762346 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:55.108265 19944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:55.108670 19944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:55.108857 19944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:55.116724 19944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:55.116909 19944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:55.117023 19944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:55.117143 19944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:55.117239 19944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:55.127918 19944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:55.129371 19944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:55.130860 19944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:55.143826 19949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:55.143786 19952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:55.143860 19950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:55.145071 19944 server_base.cc:1034] running on GCE node
I20260927 07:11:55.146046 19944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:55.147730 19944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:55.149004 19944 hybrid_clock.cc:648] HybridClock initialized: now 1790493115148973 us; error 67 us; skew 500 ppm
I20260927 07:11:55.149425 19944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:55 dist-test-slave-8vqf krb5kdc[19931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:55.152951 19944 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 07:11:55.154691 19944 webserver.cc:460] Webserver started at http://127.0.0.1:38809/ using document root <none> and password file <none>
I20260927 07:11:55.155480 19944 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:55.155647 19944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:55.155982 19944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:55.159533 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "47bcf2ca66714328b8b2e78580d7a78b"
format_stamp: "Formatted at 2026-09-27 07:11:55 on dist-test-slave-8vqf"
server_key: "528717a9f362d999cb656bbe4fe36ab7"
server_key_iv: "e0dec20650f1dd6a2e4cc0e031cd7192"
server_key_version: "encryptionkey@0"
I20260927 07:11:55.160537 19944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "47bcf2ca66714328b8b2e78580d7a78b"
format_stamp: "Formatted at 2026-09-27 07:11:55 on dist-test-slave-8vqf"
server_key: "528717a9f362d999cb656bbe4fe36ab7"
server_key_iv: "e0dec20650f1dd6a2e4cc0e031cd7192"
server_key_version: "encryptionkey@0"
I20260927 07:11:55.166456 19944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260927 07:11:55.170222 19959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:55.171829 19944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 07:11:55.172070 19944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "47bcf2ca66714328b8b2e78580d7a78b"
format_stamp: "Formatted at 2026-09-27 07:11:55 on dist-test-slave-8vqf"
server_key: "528717a9f362d999cb656bbe4fe36ab7"
server_key_iv: "e0dec20650f1dd6a2e4cc0e031cd7192"
server_key_version: "encryptionkey@0"
I20260927 07:11:55.172304 19944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:55.178625 19944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:55.183327 19944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:55.183670 19944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:55.203187 19944 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46329
I20260927 07:11:55.203230 20010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46329 every 8 connection(s)
I20260927 07:11:55.204675 19944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:55.208312 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 19944
I20260927 07:11:55.208508 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:55.208840 20011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:55.209115 30785 external_mini_cluster.cc:1428] Setting key 78ad3d83d948f3b3e14f419465c9409d
I20260927 07:11:55.217180 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:55.210899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:56119 (local address 127.0.0.1:46329)
0927 07:11:55.211610 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:55.211622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:55.211694 (+ 72us) server_negotiation.cc:407] Connection header received
0927 07:11:55.213321 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:55.213344 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:55.213729 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:55.214174 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:55.215330 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:55.215336 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:55.215359 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:55.215376 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:55.215441 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:55.215463 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:55.215714 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:55.215867 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:11:55.216069 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":280,"threads_started":1}
I20260927 07:11:55.221098 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:55.223409 20011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b: Bootstrap starting.
I20260927 07:11:55.227276 20011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:55.228372 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:55.225573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.162:40611 (local address 127.0.0.1:46329)
0927 07:11:55.225775 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:55.225781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:55.226291 (+ 510us) server_negotiation.cc:407] Connection header received
0927 07:11:55.226449 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:55.226454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:55.226524 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:55.226657 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:55.227691 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:55.227699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:55.227706 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:55.227729 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:55.227795 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:55.227820 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:55.227950 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:55.228023 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 07:11:55.228084 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 07:11:55.228746 20011 log.cc:824] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:55.231817 20011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b: No bootstrap required, opened a new log
I20260927 07:11:55.239403 20011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } }
I20260927 07:11:55.239856 20011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:55.239969 20011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bcf2ca66714328b8b2e78580d7a78b, State: Initialized, Role: FOLLOWER
I20260927 07:11:55.240587 20011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } }
I20260927 07:11:55.240844 20011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:55.241019 20011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:55.241207 20011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:55.243732 20011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } }
I20260927 07:11:55.244289 20011 leader_election.cc:302] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47bcf2ca66714328b8b2e78580d7a78b; no voters:
I20260927 07:11:55.244922 20011 leader_election.cc:288] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:55.245083 20021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:55.245759 20021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [term 1 LEADER]: Becoming Leader. State: Replica: 47bcf2ca66714328b8b2e78580d7a78b, State: Running, Role: LEADER
I20260927 07:11:55.246410 20021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } }
I20260927 07:11:55.247321 20011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:55.249729 20023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47bcf2ca66714328b8b2e78580d7a78b. Latest consensus state: current_term: 1 leader_uuid: "47bcf2ca66714328b8b2e78580d7a78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } } }
I20260927 07:11:55.249718 20022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47bcf2ca66714328b8b2e78580d7a78b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47bcf2ca66714328b8b2e78580d7a78b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46329 } } }
I20260927 07:11:55.250254 20023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:55.250617 20022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:55.252766 20029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:55.258819 20029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:55.268646 20029 catalog_manager.cc:1347] Generated new cluster ID: 8d86b155eebf4a9eb7545eac48c01b83
I20260927 07:11:55.268785 20029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:55.315716 20029 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:55.316324 20029 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:55.324895 20029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47bcf2ca66714328b8b2e78580d7a78b: Generated new TSK 0
I20260927 07:11:55.326061 20029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:55.343618 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19944
W20260927 07:11:55.345327 20006 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.162:56119
W20260927 07:11:55.346052 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 19944: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:55.346206 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 19944
2026-09-27T07:11:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:55 dist-test-slave-8vqf krb5kdc[20042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:55 dist-test-slave-8vqf krb5kdc[20042](info): set up 2 sockets
Sep 27 07:11:55 dist-test-slave-8vqf krb5kdc[20042](info): commencing operation
krb5kdc: starting...
W20260927 07:11:56.475128 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:56 dist-test-slave-8vqf krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:11:56.746405 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38375
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:38375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:57.087590 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:57.087898 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:57.088074 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:57.095422 20055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:57.095549 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:57.095595 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:57.095731 20055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:57.095853 20055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:57.106009 20055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:38375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:38375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:57.107448 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:57.108798 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:57.121202 20063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:57.121206 20061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:57.121281 20060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:57.121867 20055 server_base.cc:1034] running on GCE node
I20260927 07:11:57.122785 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:57.124424 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:57.125650 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790493117125613 us; error 62 us; skew 500 ppm
I20260927 07:11:57.126035 20055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:57 dist-test-slave-8vqf krb5kdc[20042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493117, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:57.135900 20055 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:11:57.137658 20055 webserver.cc:460] Webserver started at http://127.30.16.126:40179/ using document root <none> and password file <none>
I20260927 07:11:57.138471 20055 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:57.138656 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:57.138990 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:57.142138 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "c87c18b3bd69471ea85c44eba6e246e8"
format_stamp: "Formatted at 2026-09-27 07:11:57 on dist-test-slave-8vqf"
server_key: "537cd634f22c355b98ca0e2aaa2e8f8f"
server_key_iv: "feba91926554b7431a5ed6b09ab3827a"
server_key_version: "encryptionkey@0"
I20260927 07:11:57.143023 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "c87c18b3bd69471ea85c44eba6e246e8"
format_stamp: "Formatted at 2026-09-27 07:11:57 on dist-test-slave-8vqf"
server_key: "537cd634f22c355b98ca0e2aaa2e8f8f"
server_key_iv: "feba91926554b7431a5ed6b09ab3827a"
server_key_version: "encryptionkey@0"
I20260927 07:11:57.148427 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:11:57.152143 20070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:57.153571 20055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:11:57.153769 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "c87c18b3bd69471ea85c44eba6e246e8"
format_stamp: "Formatted at 2026-09-27 07:11:57 on dist-test-slave-8vqf"
server_key: "537cd634f22c355b98ca0e2aaa2e8f8f"
server_key_iv: "feba91926554b7431a5ed6b09ab3827a"
server_key_version: "encryptionkey@0"
I20260927 07:11:57.153981 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:57.165681 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:57.170337 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:57.170711 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:57.189915 20055 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:38375
I20260927 07:11:57.189935 20121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:38375 every 8 connection(s)
I20260927 07:11:57.191475 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:57.192008 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20055
I20260927 07:11:57.192200 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:57.192891 30785 external_mini_cluster.cc:1428] Setting key 7956fc1ed8061f71b2e024008004a5a5
I20260927 07:11:57.196667 20122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:11:57 dist-test-slave-8vqf krb5kdc[20042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:11:57.209776 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8: Bootstrap starting.
I20260927 07:11:57.214293 20122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:57.215776 20122 log.cc:824] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:57.218430 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:57.194875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60413 (local address 127.30.16.126:38375)
0927 07:11:57.195687 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:57.195705 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:57.195758 (+ 53us) server_negotiation.cc:407] Connection header received
0927 07:11:57.198128 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:57.198158 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:57.198743 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:57.199196 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:57.200499 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:57.202687 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:57.203997 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:57.205052 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:57.205236 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:57.209645 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:57.209659 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:57.209681 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:57.209729 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:57.213742 (+ 4013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:57.214652 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:57.214664 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:57.214674 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:57.214919 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:57.215348 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:57.215353 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:57.215356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:57.216129 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:57.216534 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:57.216975 (+ 441us) server_negotiation.cc:299] Negotiation successful
0927 07:11:57.217336 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":323,"threads_started":1}
I20260927 07:11:57.219135 20122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8: No bootstrap required, opened a new log
I20260927 07:11:57.222617 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:57.230307 20122 raft_consensus.cc:348] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } }
I20260927 07:11:57.230839 20122 raft_consensus.cc:374] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:57.230965 20122 raft_consensus.cc:729] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87c18b3bd69471ea85c44eba6e246e8, State: Initialized, Role: FOLLOWER
I20260927 07:11:57.231752 20122 consensus_queue.cc:260] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } }
I20260927 07:11:57.232018 20122 raft_consensus.cc:388] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:57.232189 20122 raft_consensus.cc:482] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:57.232409 20122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:57.235251 20122 raft_consensus.cc:504] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } }
I20260927 07:11:57.235831 20122 leader_election.cc:302] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c87c18b3bd69471ea85c44eba6e246e8; no voters:
I20260927 07:11:57.236554 20122 leader_election.cc:288] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:57.236725 20132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:57.237684 20132 raft_consensus.cc:686] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [term 1 LEADER]: Becoming Leader. State: Replica: c87c18b3bd69471ea85c44eba6e246e8, State: Running, Role: LEADER
I20260927 07:11:57.238292 20132 consensus_queue.cc:237] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } }
I20260927 07:11:57.239172 20122 sys_catalog.cc:564] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:57.241803 20134 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c87c18b3bd69471ea85c44eba6e246e8. Latest consensus state: current_term: 1 leader_uuid: "c87c18b3bd69471ea85c44eba6e246e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } } }
I20260927 07:11:57.241860 20133 sys_catalog.cc:455] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c87c18b3bd69471ea85c44eba6e246e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c87c18b3bd69471ea85c44eba6e246e8" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 38375 } } }
I20260927 07:11:57.242223 20134 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:57.242288 20133 sys_catalog.cc:458] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8 [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:57.243607 20138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:57.246053 20125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:57.228392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52769 (local address 127.30.16.126:38375)
0927 07:11:57.228627 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:57.228634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:57.228950 (+ 316us) server_negotiation.cc:407] Connection header received
0927 07:11:57.229107 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:57.229113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:57.229206 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:57.229349 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:57.230880 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:57.231780 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:57.233233 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:57.233715 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:57.233874 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:57.236680 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:57.236686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:57.236691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:11:57.236747 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:57.241128 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:57.242397 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:57.242403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:57.242408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:57.242643 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:11:57.243337 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:11:57.243345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:11:57.243349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:11:57.243748 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:57.245070 (+ 1322us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:57.245251 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 07:11:57.245476 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:11:57.260629 20138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:57.273102 20138 catalog_manager.cc:1347] Generated new cluster ID: e1ece09e696c4172a90d573e248aff10
I20260927 07:11:57.273252 20138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:57.288677 20138 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:57.290689 20138 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:57.302462 20138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c87c18b3bd69471ea85c44eba6e246e8: Generated new TSK 0
I20260927 07:11:57.303633 20138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:57.317688 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20055
I20260927 07:11:57.319218 20117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:11:57.417236 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20055
W20260927 07:11:57.431962 20040 connection.cc:535] client connection to 127.30.16.126:38375 recv error: Network error: failed to read from TLS socket (remote: 127.30.16.126:38375): Connection reset by peer (error 104)
2026-09-27T07:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:57 dist-test-slave-8vqf krb5kdc[20154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:57 dist-test-slave-8vqf krb5kdc[20154](info): set up 2 sockets
Sep 27 07:11:57 dist-test-slave-8vqf krb5kdc[20154](info): commencing operation
krb5kdc: starting...
W20260927 07:11:58.535885 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:11:58 dist-test-slave-8vqf krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:11:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:11:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:11:58.806196 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33419
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:11:59.148365 20167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:11:59.148672 20167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:11:59.148867 20167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:11:59.156323 20167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:11:59.156452 20167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:11:59.156579 20167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:11:59.156695 20167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:11:59.156757 20167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:11:59.167002 20167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:33419
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:11:59.168416 20167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:11:59.169994 20167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:11:59.182446 20172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:59.182421 20173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:11:59.182477 20175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:11:59.183970 20167 server_base.cc:1034] running on GCE node
I20260927 07:11:59.184839 20167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:11:59.186407 20167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:11:59.187645 20167 hybrid_clock.cc:648] HybridClock initialized: now 1790493119187594 us; error 75 us; skew 500 ppm
I20260927 07:11:59.188030 20167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:11:59 dist-test-slave-8vqf krb5kdc[20154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493119, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:11:59.191294 20167 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:11:59.192869 20167 webserver.cc:460] Webserver started at http://127.30.16.126:40415/ using document root <none> and password file <none>
I20260927 07:11:59.193619 20167 fs_manager.cc:359] Metadata directory not provided
I20260927 07:11:59.193778 20167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:11:59.194116 20167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:11:59.197438 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "723c45532ba1466fa11f6ec0b674133d"
format_stamp: "Formatted at 2026-09-27 07:11:59 on dist-test-slave-8vqf"
server_key: "1e9ed4cda5c87ef37a423e26ce43a00e"
server_key_iv: "b3fae40a40923ab1e6333f8e5c362e49"
server_key_version: "encryptionkey@0"
I20260927 07:11:59.198313 20167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "723c45532ba1466fa11f6ec0b674133d"
format_stamp: "Formatted at 2026-09-27 07:11:59 on dist-test-slave-8vqf"
server_key: "1e9ed4cda5c87ef37a423e26ce43a00e"
server_key_iv: "b3fae40a40923ab1e6333f8e5c362e49"
server_key_version: "encryptionkey@0"
I20260927 07:11:59.204174 20167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 07:11:59.207897 20182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:11:59.209370 20167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 07:11:59.209568 20167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "723c45532ba1466fa11f6ec0b674133d"
format_stamp: "Formatted at 2026-09-27 07:11:59 on dist-test-slave-8vqf"
server_key: "1e9ed4cda5c87ef37a423e26ce43a00e"
server_key_iv: "b3fae40a40923ab1e6333f8e5c362e49"
server_key_version: "encryptionkey@0"
I20260927 07:11:59.209785 20167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:11:59.229400 20167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:11:59.233719 20167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:11:59.234048 20167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:11:59.253158 20167 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:33419
I20260927 07:11:59.253181 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:33419 every 8 connection(s)
I20260927 07:11:59.254751 20167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:11:59.258659 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:11:59.263221 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20167
I20260927 07:11:59.263422 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:11:59.263974 30785 external_mini_cluster.cc:1428] Setting key 34b4fee78fe254d95068140ce4698a24
I20260927 07:11:59.270762 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d: Bootstrap starting.
I20260927 07:11:59.274226 20234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d: Neither blocks nor log segments found. Creating new log.
I20260927 07:11:59.275568 20234 log.cc:824] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d: Log is configured to *not* fsync() on all Append() calls
I20260927 07:11:59.279023 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:59.265970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43611 (local address 127.30.16.126:33419)
0927 07:11:59.266802 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:59.266815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:59.266858 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:11:59.269197 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:59.269222 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:59.269811 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:59.270237 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:59.270931 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:59.272246 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:59.273643 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:59.274471 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:59.274726 (+ 255us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:59.276211 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:59.276228 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:59.276259 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:59.277097 (+ 838us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:59.277203 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:59.277232 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:59.277413 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:59.277711 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 07:11:59.277963 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":339,"threads_started":1}
I20260927 07:11:59.279066 20234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d: No bootstrap required, opened a new log
I20260927 07:11:59.283197 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:11:59.288002 20234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } }
I20260927 07:11:59.288527 20234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:11:59.288683 20234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723c45532ba1466fa11f6ec0b674133d, State: Initialized, Role: FOLLOWER
I20260927 07:11:59.289399 20234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } }
I20260927 07:11:59.289697 20234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:11:59.289837 20234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:11:59.290022 20234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:11:59.292886 20234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } }
I20260927 07:11:59.293484 20234 leader_election.cc:302] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 723c45532ba1466fa11f6ec0b674133d; no voters:
I20260927 07:11:59.294689 20234 leader_election.cc:288] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:11:59.294688 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:11:59.287582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57131 (local address 127.30.16.126:33419)
0927 07:11:59.287784 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 07:11:59.287790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:11:59.288143 (+ 353us) server_negotiation.cc:407] Connection header received
0927 07:11:59.288277 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:11:59.288282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:11:59.288361 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:11:59.288483 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:11:59.289103 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:59.290271 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:59.291782 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:11:59.292491 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:11:59.292633 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:11:59.293718 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:11:59.293724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:11:59.293729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:11:59.293750 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:11:59.293817 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:11:59.293841 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:11:59.293982 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 07:11:59.294122 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 07:11:59.294193 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:11:59.294824 20244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:11:59.297170 20234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:11:59.301898 20244 raft_consensus.cc:686] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [term 1 LEADER]: Becoming Leader. State: Replica: 723c45532ba1466fa11f6ec0b674133d, State: Running, Role: LEADER
I20260927 07:11:59.302501 20244 consensus_queue.cc:237] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } }
I20260927 07:11:59.305840 20245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "723c45532ba1466fa11f6ec0b674133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } } }
I20260927 07:11:59.306258 20245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:59.306823 20246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 723c45532ba1466fa11f6ec0b674133d. Latest consensus state: current_term: 1 leader_uuid: "723c45532ba1466fa11f6ec0b674133d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "723c45532ba1466fa11f6ec0b674133d" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 33419 } } }
I20260927 07:11:59.307111 20246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d [sys.catalog]: This master's current role is: LEADER
I20260927 07:11:59.309427 20258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:11:59.314153 20258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:11:59.323662 20258 catalog_manager.cc:1347] Generated new cluster ID: 61844e95b12a4f569d587016f1989ecb
I20260927 07:11:59.323801 20258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:11:59.334777 20258 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:11:59.336719 20258 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:11:59.347287 20258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 723c45532ba1466fa11f6ec0b674133d: Generated new TSK 0
I20260927 07:11:59.348387 20258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:11:59.364434 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20167
W20260927 07:11:59.365854 20229 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43611
W20260927 07:11:59.366757 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20167: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:11:59.366926 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20167
W20260927 07:11:59.384179 20152 connection.cc:535] client connection to 127.30.16.126:33419 recv error: Network error: failed to read from TLS socket (remote: 127.30.16.126:33419): Connection reset by peer (error 104)
2026-09-27T07:11:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:11:59 dist-test-slave-8vqf krb5kdc[20265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:11:59 dist-test-slave-8vqf krb5kdc[20265](info): set up 2 sockets
Sep 27 07:11:59 dist-test-slave-8vqf krb5kdc[20265](info): commencing operation
krb5kdc: starting...
W20260927 07:12:00.483180 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:00 dist-test-slave-8vqf krb5kdc[20265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:00.762678 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44665
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:44665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:01.104602 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:01.104923 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:01.105086 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:01.112632 20278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:01.112761 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:01.112875 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:01.112982 20278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:01.113070 20278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:01.123502 20278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:44665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:44665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:01.124903 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:01.126272 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:01.138298 20283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:01.138355 20284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:01.138351 20286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:01.139477 20278 server_base.cc:1034] running on GCE node
I20260927 07:12:01.140362 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:01.141922 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:01.143163 20278 hybrid_clock.cc:648] HybridClock initialized: now 1790493121143119 us; error 40 us; skew 500 ppm
I20260927 07:12:01.143549 20278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:01 dist-test-slave-8vqf krb5kdc[20265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493121, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:01.146867 20278 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:01.148399 20278 webserver.cc:460] Webserver started at http://127.30.16.126:35111/ using document root <none> and password file <none>
I20260927 07:12:01.149070 20278 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:01.149171 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:01.149444 20278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:01.152549 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "3e3e4fbd8e1942bd9e153596ee55caa6"
format_stamp: "Formatted at 2026-09-27 07:12:01 on dist-test-slave-8vqf"
server_key: "5a25402f7f922c945fec7ab9e5dcacff"
server_key_iv: "544814c3b53ae0b3490013cbfdbe8af3"
server_key_version: "encryptionkey@0"
I20260927 07:12:01.153296 20278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "3e3e4fbd8e1942bd9e153596ee55caa6"
format_stamp: "Formatted at 2026-09-27 07:12:01 on dist-test-slave-8vqf"
server_key: "5a25402f7f922c945fec7ab9e5dcacff"
server_key_iv: "544814c3b53ae0b3490013cbfdbe8af3"
server_key_version: "encryptionkey@0"
I20260927 07:12:01.158883 20278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 07:12:01.162515 20293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:01.164042 20278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:01.164258 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "3e3e4fbd8e1942bd9e153596ee55caa6"
format_stamp: "Formatted at 2026-09-27 07:12:01 on dist-test-slave-8vqf"
server_key: "5a25402f7f922c945fec7ab9e5dcacff"
server_key_iv: "544814c3b53ae0b3490013cbfdbe8af3"
server_key_version: "encryptionkey@0"
I20260927 07:12:01.164485 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:01.170645 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:01.174947 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:01.175253 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:01.194327 20278 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:44665
I20260927 07:12:01.194350 20344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:44665 every 8 connection(s)
I20260927 07:12:01.195899 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:01.198626 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20278
I20260927 07:12:01.198851 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:01.199308 30785 external_mini_cluster.cc:1428] Setting key 700f6a0555b806be75c65093cff686d5
I20260927 07:12:01.200277 20345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:01.207261 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:01.201110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42377 (local address 127.30.16.126:44665)
0927 07:12:01.201951 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:01.201964 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:01.202005 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:12:01.203675 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:01.203697 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:01.204073 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:01.204437 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:01.205524 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:01.205531 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:01.205544 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:01.205559 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:01.205636 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:01.205658 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:01.205834 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:01.205951 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 07:12:01.206183 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":369,"threads_started":1}
I20260927 07:12:01.211160 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:01.213929 20345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6: Bootstrap starting.
I20260927 07:12:01.217823 20345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:01.218238 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:01.215614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33373 (local address 127.30.16.126:44665)
0927 07:12:01.215781 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:01.215786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:01.216245 (+ 459us) server_negotiation.cc:407] Connection header received
0927 07:12:01.216453 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:01.216463 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:01.216513 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:01.216619 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:01.217572 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:01.217577 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:01.217580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:01.217595 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:01.217635 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:01.217650 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:01.217769 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:01.217890 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 07:12:01.217953 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 07:12:01.219216 20345 log.cc:824] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:01.222041 20345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6: No bootstrap required, opened a new log
I20260927 07:12:01.230007 20345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } }
I20260927 07:12:01.230441 20345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:12:01.230553 20345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3e4fbd8e1942bd9e153596ee55caa6, State: Initialized, Role: FOLLOWER
I20260927 07:12:01.231240 20345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } }
I20260927 07:12:01.231484 20345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:12:01.231628 20345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:12:01.231830 20345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:01.234239 20345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } }
I20260927 07:12:01.234855 20345 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3e4fbd8e1942bd9e153596ee55caa6; no voters:
I20260927 07:12:01.235494 20345 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:12:01.235690 20355 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:01.236480 20355 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3e4fbd8e1942bd9e153596ee55caa6, State: Running, Role: LEADER
I20260927 07:12:01.237038 20355 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } }
I20260927 07:12:01.237643 20345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:01.239739 20357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3e4fbd8e1942bd9e153596ee55caa6. Latest consensus state: current_term: 1 leader_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } } }
I20260927 07:12:01.240155 20357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:01.240701 20356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3e4fbd8e1942bd9e153596ee55caa6" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 44665 } } }
I20260927 07:12:01.241427 20356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:01.243376 20361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:12:01.249349 20361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:12:01.259652 20361 catalog_manager.cc:1347] Generated new cluster ID: ee33d4f1b3de4987b840f7f06d5bbd1a
I20260927 07:12:01.259793 20361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:01.278451 20361 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:01.278998 20361 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:01.286899 20361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3e4fbd8e1942bd9e153596ee55caa6: Generated new TSK 0
I20260927 07:12:01.287755 20361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:01.296190 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20278
W20260927 07:12:01.297435 20340 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42377
W20260927 07:12:01.298295 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20278: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:12:01.298462 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20278
2026-09-27T07:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:12:01 dist-test-slave-8vqf krb5kdc[20377](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:12:01 dist-test-slave-8vqf krb5kdc[20377](info): set up 2 sockets
Sep 27 07:12:01 dist-test-slave-8vqf krb5kdc[20377](info): commencing operation
krb5kdc: starting...
W20260927 07:12:02.415175 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:02 dist-test-slave-8vqf krb5kdc[20377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:02.693981 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:45937
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:45937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:03.050520 20390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:03.050911 20390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:03.051009 20390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:03.058454 20390 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:03.058668 20390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:03.058743 20390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:03.058811 20390 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:03.058856 20390 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:03.069298 20390 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:45937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:45937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:03.070813 20390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:03.072134 20390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:03.084600 20395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:03.084661 20396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:03.084682 20398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:03.085634 20390 server_base.cc:1034] running on GCE node
I20260927 07:12:03.086486 20390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:03.088040 20390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:03.089279 20390 hybrid_clock.cc:648] HybridClock initialized: now 1790493123089233 us; error 37 us; skew 500 ppm
I20260927 07:12:03.089663 20390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:03 dist-test-slave-8vqf krb5kdc[20377](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493123, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:03.098183 20390 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:03.100103 20390 webserver.cc:460] Webserver started at http://127.30.16.126:41105/ using document root <none> and password file <none>
I20260927 07:12:03.100924 20390 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:03.101082 20390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:03.101414 20390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:03.104797 20390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "9cc1907ac09549d39447b7749f28c8cf"
format_stamp: "Formatted at 2026-09-27 07:12:03 on dist-test-slave-8vqf"
server_key: "83fcd1ac3094f1eb30611384bb29ef8c"
server_key_iv: "434282c965b05a890c07e52803ac474f"
server_key_version: "encryptionkey@0"
I20260927 07:12:03.105638 20390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "9cc1907ac09549d39447b7749f28c8cf"
format_stamp: "Formatted at 2026-09-27 07:12:03 on dist-test-slave-8vqf"
server_key: "83fcd1ac3094f1eb30611384bb29ef8c"
server_key_iv: "434282c965b05a890c07e52803ac474f"
server_key_version: "encryptionkey@0"
I20260927 07:12:03.111310 20390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:12:03.115020 20405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:03.116444 20390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 07:12:03.116647 20390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "9cc1907ac09549d39447b7749f28c8cf"
format_stamp: "Formatted at 2026-09-27 07:12:03 on dist-test-slave-8vqf"
server_key: "83fcd1ac3094f1eb30611384bb29ef8c"
server_key_iv: "434282c965b05a890c07e52803ac474f"
server_key_version: "encryptionkey@0"
I20260927 07:12:03.116865 20390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:03.132813 20390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:03.137389 20390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:03.137705 20390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:03.157522 20390 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:45937
I20260927 07:12:03.157529 20456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:45937 every 8 connection(s)
I20260927 07:12:03.159019 20390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:03.161824 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20390
I20260927 07:12:03.162026 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:03.162474 30785 external_mini_cluster.cc:1428] Setting key a9d6fb861abedbc11a4b39ae9103c5a6
I20260927 07:12:03.163213 20457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 07:12:03 dist-test-slave-8vqf krb5kdc[20377](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:12:03.179416 20457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf: Bootstrap starting.
I20260927 07:12:03.183478 20457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:03.184698 20457 log.cc:824] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:03.187307 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:03.164375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60995 (local address 127.30.16.126:45937)
0927 07:12:03.165181 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:03.165194 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:03.165236 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:12:03.166887 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:03.166912 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:03.167365 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:03.167727 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:03.169620 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:03.171137 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:03.172911 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:03.173629 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:03.173765 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:03.179450 (+ 5685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:03.179460 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:03.179478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:03.179521 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:03.182910 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:03.183650 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:03.183667 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:03.183673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:03.183849 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:03.184256 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:03.184261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:03.184264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:03.184892 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:03.185213 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:03.185700 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 07:12:03.186149 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":369,"threads_started":1}
I20260927 07:12:03.188194 20457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf: No bootstrap required, opened a new log
I20260927 07:12:03.191236 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:03.197165 20457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } }
I20260927 07:12:03.197744 20457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:12:03.197875 20457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc1907ac09549d39447b7749f28c8cf, State: Initialized, Role: FOLLOWER
I20260927 07:12:03.198700 20457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } }
I20260927 07:12:03.199023 20457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:12:03.199218 20457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:12:03.199447 20457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:03.202966 20457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } }
I20260927 07:12:03.203747 20457 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc1907ac09549d39447b7749f28c8cf; no voters:
I20260927 07:12:03.204516 20457 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:12:03.204588 20467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:03.207297 20457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:03.207072 20467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [term 1 LEADER]: Becoming Leader. State: Replica: 9cc1907ac09549d39447b7749f28c8cf, State: Running, Role: LEADER
I20260927 07:12:03.208357 20467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } }
I20260927 07:12:03.211970 20468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc1907ac09549d39447b7749f28c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } } }
I20260927 07:12:03.212358 20468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:03.214418 20469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc1907ac09549d39447b7749f28c8cf. Latest consensus state: current_term: 1 leader_uuid: "9cc1907ac09549d39447b7749f28c8cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc1907ac09549d39447b7749f28c8cf" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 45937 } } }
I20260927 07:12:03.215508 20469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:03.217289 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:03.195773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56723 (local address 127.30.16.126:45937)
0927 07:12:03.195993 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:03.196000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:03.196604 (+ 604us) server_negotiation.cc:407] Connection header received
0927 07:12:03.196822 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:03.196825 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:03.196890 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:03.196970 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:03.199192 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:03.200031 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:03.201903 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:03.202349 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:03.202458 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:03.205940 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:03.205947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:03.205952 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:03.206001 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:03.211059 (+ 5058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:03.212401 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:03.212410 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:03.212417 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:03.212713 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:03.213341 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:03.213350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:03.213357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:03.213928 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:03.214314 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:03.216256 (+ 1942us) server_negotiation.cc:299] Negotiation successful
0927 07:12:03.216617 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:12:03.220790 20477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:12:03.226035 20477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:12:03.238124 20477 catalog_manager.cc:1347] Generated new cluster ID: c35e3b0408b84ce8a42a3f98780cd1fc
I20260927 07:12:03.238291 20477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:03.264711 20477 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:03.266906 20477 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:03.280217 20477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc1907ac09549d39447b7749f28c8cf: Generated new TSK 0
I20260927 07:12:03.281519 20477 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:03.291432 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20390
I20260927 07:12:03.292712 20452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:03.388100 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20390
2026-09-27T07:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2091 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:12:03 dist-test-slave-8vqf krb5kdc[20490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:12:03 dist-test-slave-8vqf krb5kdc[20490](info): set up 2 sockets
Sep 27 07:12:03 dist-test-slave-8vqf krb5kdc[20490](info): commencing operation
krb5kdc: starting...
W20260927 07:12:04.571218 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:04 dist-test-slave-8vqf krb5kdc[20490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:04.838140 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40785
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:40785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:05.185393 20503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:05.185745 20503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:05.185920 20503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:05.193507 20503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:05.193642 20503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:05.193760 20503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:05.193876 20503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:05.193955 20503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:05.204409 20503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:40785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:40785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:05.205832 20503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:05.207284 20503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:05.223196 20509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:05.223166 20508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:05.223217 20511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:05.224206 20503 server_base.cc:1034] running on GCE node
I20260927 07:12:05.224984 20503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:05.226519 20503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:05.227885 20503 hybrid_clock.cc:648] HybridClock initialized: now 1790493125227825 us; error 47 us; skew 500 ppm
I20260927 07:12:05.228402 20503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:05 dist-test-slave-8vqf krb5kdc[20490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493125, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:05.232018 20503 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:05.233795 20503 webserver.cc:460] Webserver started at http://127.30.16.126:36167/ using document root <none> and password file <none>
I20260927 07:12:05.234627 20503 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:05.234793 20503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:05.235126 20503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:05.238358 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "f3b99cf041ef4b13adeb99acffada671"
format_stamp: "Formatted at 2026-09-27 07:12:05 on dist-test-slave-8vqf"
server_key: "7f7242ce41fe8362a3d52dde46876b6b"
server_key_iv: "562fbeee64e33168c9f1f877801d1613"
server_key_version: "encryptionkey@0"
I20260927 07:12:05.239318 20503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "f3b99cf041ef4b13adeb99acffada671"
format_stamp: "Formatted at 2026-09-27 07:12:05 on dist-test-slave-8vqf"
server_key: "7f7242ce41fe8362a3d52dde46876b6b"
server_key_iv: "562fbeee64e33168c9f1f877801d1613"
server_key_version: "encryptionkey@0"
I20260927 07:12:05.245122 20503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 07:12:05.249135 20518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:05.250617 20503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 07:12:05.250811 20503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "f3b99cf041ef4b13adeb99acffada671"
format_stamp: "Formatted at 2026-09-27 07:12:05 on dist-test-slave-8vqf"
server_key: "7f7242ce41fe8362a3d52dde46876b6b"
server_key_iv: "562fbeee64e33168c9f1f877801d1613"
server_key_version: "encryptionkey@0"
I20260927 07:12:05.251022 20503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:05.274747 20503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:05.279023 20503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:05.279335 20503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:05.298414 20503 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:40785
I20260927 07:12:05.298437 20569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:40785 every 8 connection(s)
I20260927 07:12:05.299947 20503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:05.304419 20570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:05.304648 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20503
I20260927 07:12:05.304832 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:05.305431 30785 external_mini_cluster.cc:1428] Setting key 555868e46bd4a94889ff07f46cad4141
I20260927 07:12:05.318796 20570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671: Bootstrap starting.
I20260927 07:12:05.319191 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:05.307416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56581 (local address 127.30.16.126:40785)
0927 07:12:05.308251 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:05.308263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:05.308304 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:12:05.310028 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:05.310052 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:05.310476 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:05.310996 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:05.311595 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:05.312923 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:05.314255 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:05.315100 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:05.315346 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:05.316631 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:05.316640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:05.316656 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:05.317241 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:05.317321 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:05.317346 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:05.317487 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:05.317718 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 07:12:05.318035 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":382,"threads_started":1}
I20260927 07:12:05.322350 20570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:05.323105 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:05.323643 20570 log.cc:824] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:05.326803 20570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671: No bootstrap required, opened a new log
I20260927 07:12:05.334252 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:05.327529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40761 (local address 127.30.16.126:40785)
0927 07:12:05.327791 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:05.327798 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:05.328104 (+ 306us) server_negotiation.cc:407] Connection header received
0927 07:12:05.328304 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:05.328308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:05.328370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:05.328453 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:05.329095 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:05.329812 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:05.331704 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:05.332184 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:05.332282 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:05.333435 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:05.333439 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:05.333443 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:05.333460 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:05.333499 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:05.333516 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:05.333584 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:05.333742 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 07:12:05.333814 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:12:05.336793 20570 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } }
I20260927 07:12:05.337195 20570 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:12:05.337320 20570 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3b99cf041ef4b13adeb99acffada671, State: Initialized, Role: FOLLOWER
I20260927 07:12:05.337970 20570 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } }
I20260927 07:12:05.338214 20570 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:12:05.338370 20570 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:12:05.338553 20570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:05.340834 20570 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } }
I20260927 07:12:05.341358 20570 leader_election.cc:302] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3b99cf041ef4b13adeb99acffada671; no voters:
I20260927 07:12:05.341951 20570 leader_election.cc:288] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:12:05.342113 20580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:05.342859 20580 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [term 1 LEADER]: Becoming Leader. State: Replica: f3b99cf041ef4b13adeb99acffada671, State: Running, Role: LEADER
I20260927 07:12:05.343412 20580 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } }
I20260927 07:12:05.344173 20570 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:05.346095 20581 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3b99cf041ef4b13adeb99acffada671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } } }
I20260927 07:12:05.346612 20581 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:05.346760 20582 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3b99cf041ef4b13adeb99acffada671. Latest consensus state: current_term: 1 leader_uuid: "f3b99cf041ef4b13adeb99acffada671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3b99cf041ef4b13adeb99acffada671" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 40785 } } }
I20260927 07:12:05.347086 20582 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:05.347751 20585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:12:05.354883 20585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:12:05.364638 20585 catalog_manager.cc:1347] Generated new cluster ID: 82847130dd214a8ebb9cdd1fcb117849
I20260927 07:12:05.364789 20585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:05.395473 20585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:05.397535 20585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:05.406533 20585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3b99cf041ef4b13adeb99acffada671: Generated new TSK 0
I20260927 07:12:05.407697 20585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:05.413050 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20503
W20260927 07:12:05.414769 20565 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56581
W20260927 07:12:05.415560 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20503: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:12:05.415738 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20503
2026-09-27T07:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:12:05 dist-test-slave-8vqf krb5kdc[20601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:12:05 dist-test-slave-8vqf krb5kdc[20601](info): set up 2 sockets
Sep 27 07:12:05 dist-test-slave-8vqf krb5kdc[20601](info): commencing operation
krb5kdc: starting...
W20260927 07:12:06.555182 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:06 dist-test-slave-8vqf krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:06.826798 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:41409
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:07.173995 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:07.174405 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:07.174639 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:07.182184 20614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:07.182379 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:07.182471 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:07.182588 20614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:07.182662 20614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:07.193198 20614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:41409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:07.194723 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:07.196172 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:07.208938 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:07.209023 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:07.208976 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:07.209997 20614 server_base.cc:1034] running on GCE node
I20260927 07:12:07.211004 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:07.212574 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:07.213828 20614 hybrid_clock.cc:648] HybridClock initialized: now 1790493127213770 us; error 53 us; skew 500 ppm
I20260927 07:12:07.214238 20614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:07 dist-test-slave-8vqf krb5kdc[20601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493127, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:07.217805 20614 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:07.219437 20614 webserver.cc:460] Webserver started at http://127.30.16.126:35035/ using document root <none> and password file <none>
I20260927 07:12:07.220165 20614 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:07.220316 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:07.220652 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:07.223919 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "e8a6685e018e4c6e8caad2985348d069"
format_stamp: "Formatted at 2026-09-27 07:12:07 on dist-test-slave-8vqf"
server_key: "b604be0f05f09b33e81d0704adaaac5a"
server_key_iv: "d372183486863fcc0e212c8b5f0a9447"
server_key_version: "encryptionkey@0"
I20260927 07:12:07.224716 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "e8a6685e018e4c6e8caad2985348d069"
format_stamp: "Formatted at 2026-09-27 07:12:07 on dist-test-slave-8vqf"
server_key: "b604be0f05f09b33e81d0704adaaac5a"
server_key_iv: "d372183486863fcc0e212c8b5f0a9447"
server_key_version: "encryptionkey@0"
I20260927 07:12:07.230247 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 07:12:07.234066 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:07.235647 20614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:07.235798 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "e8a6685e018e4c6e8caad2985348d069"
format_stamp: "Formatted at 2026-09-27 07:12:07 on dist-test-slave-8vqf"
server_key: "b604be0f05f09b33e81d0704adaaac5a"
server_key_iv: "d372183486863fcc0e212c8b5f0a9447"
server_key_version: "encryptionkey@0"
I20260927 07:12:07.235987 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:07.241883 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:07.246085 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:07.246332 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:07.265452 20614 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:41409
I20260927 07:12:07.265483 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:41409 every 8 connection(s)
I20260927 07:12:07.267009 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:07.267494 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20614
I20260927 07:12:07.267690 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:07.268368 30785 external_mini_cluster.cc:1428] Setting key 9c2e94252fdab119c2372d2e87808670
I20260927 07:12:07.271948 20681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:07.276463 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:07.270269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35799 (local address 127.30.16.126:41409)
0927 07:12:07.271068 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:07.271081 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:07.271123 (+ 42us) server_negotiation.cc:407] Connection header received
0927 07:12:07.272862 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:07.272888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:07.273228 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:07.273623 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:07.274730 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:07.274739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:07.274756 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:07.274776 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:07.274849 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:07.274871 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:07.275129 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:07.275270 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 07:12:07.275461 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":369,"threads_started":1}
I20260927 07:12:07.280244 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:07.283820 20681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069: Bootstrap starting.
I20260927 07:12:07.287169 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:07.284625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54661 (local address 127.30.16.126:41409)
0927 07:12:07.284815 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:07.284824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:07.285163 (+ 339us) server_negotiation.cc:407] Connection header received
0927 07:12:07.285379 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:07.285383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:07.285436 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:07.285543 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:07.286459 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:07.286466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:07.286475 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 07:12:07.286490 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:07.286528 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 07:12:07.286596 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:07.286721 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:07.286840 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 07:12:07.286910 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 07:12:07.287750 20681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:07.289004 20681 log.cc:824] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:07.291862 20681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069: No bootstrap required, opened a new log
I20260927 07:12:07.299631 20681 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } }
I20260927 07:12:07.300093 20681 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 07:12:07.300205 20681 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a6685e018e4c6e8caad2985348d069, State: Initialized, Role: FOLLOWER
I20260927 07:12:07.300976 20681 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } }
I20260927 07:12:07.301263 20681 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 07:12:07.301425 20681 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 07:12:07.301608 20681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:07.303954 20681 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } }
I20260927 07:12:07.304497 20681 leader_election.cc:302] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8a6685e018e4c6e8caad2985348d069; no voters:
I20260927 07:12:07.305115 20681 leader_election.cc:288] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 07:12:07.305263 20691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:07.306416 20691 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [term 1 LEADER]: Becoming Leader. State: Replica: e8a6685e018e4c6e8caad2985348d069, State: Running, Role: LEADER
I20260927 07:12:07.307092 20691 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } }
I20260927 07:12:07.307581 20681 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:07.310211 20693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a6685e018e4c6e8caad2985348d069. Latest consensus state: current_term: 1 leader_uuid: "e8a6685e018e4c6e8caad2985348d069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } } }
I20260927 07:12:07.310775 20693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:07.310192 20692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8a6685e018e4c6e8caad2985348d069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a6685e018e4c6e8caad2985348d069" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 41409 } } }
I20260927 07:12:07.311188 20692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:07.311710 20698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:12:07.317000 20698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:12:07.330418 20698 catalog_manager.cc:1347] Generated new cluster ID: efcaa497ef8e4b529192a73a49404f95
I20260927 07:12:07.330595 20698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:07.344817 20698 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:07.345312 20698 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:07.356506 20698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a6685e018e4c6e8caad2985348d069: Generated new TSK 0
I20260927 07:12:07.357506 20698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:07.364308 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20614
W20260927 07:12:07.365552 20676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35799
W20260927 07:12:07.366382 30785 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 07:12:07.366535 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20614
2026-09-27T07:12:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1951 ms)
[----------] 18 tests from AuthTokenIssuingTest (35842 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:12:07 dist-test-slave-8vqf krb5kdc[20712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:12:07 dist-test-slave-8vqf krb5kdc[20712](info): set up 2 sockets
Sep 27 07:12:07 dist-test-slave-8vqf krb5kdc[20712](info): commencing operation
krb5kdc: starting...
W20260927 07:12:08.475435 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:08 dist-test-slave-8vqf krb5kdc[20712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:08.755292 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37481
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:09.091922 20725 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:12:09.092320 20725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:09.092463 20725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:09.092622 20725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:09.100013 20725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:09.100106 20725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:09.100220 20725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:09.100308 20725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:09.100407 20725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:09.110566 20725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:37481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:09.111984 20725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:09.113363 20725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:09.125572 20730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:09.125572 20733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:09.125597 20731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:09.128446 20725 server_base.cc:1034] running on GCE node
I20260927 07:12:09.129276 20725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:09.130854 20725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:09.132100 20725 hybrid_clock.cc:648] HybridClock initialized: now 1790493129132026 us; error 56 us; skew 500 ppm
I20260927 07:12:09.132455 20725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:09.140996 20725 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:09.142776 20725 webserver.cc:460] Webserver started at http://127.30.16.126:44021/ using document root <none> and password file <none>
I20260927 07:12:09.143519 20725 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:09.143612 20725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:09.143880 20725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:09.147050 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "a5dbc41877a0441a8106567650951b3f"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "41a553feabcbb26b8650867c8f00d626"
server_key_iv: "f02c1a9fcde217c302f7b8ee46c2b7b3"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.147825 20725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "a5dbc41877a0441a8106567650951b3f"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "41a553feabcbb26b8650867c8f00d626"
server_key_iv: "f02c1a9fcde217c302f7b8ee46c2b7b3"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.153563 20725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:12:09.157337 20740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:09.158855 20725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:09.159010 20725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "a5dbc41877a0441a8106567650951b3f"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "41a553feabcbb26b8650867c8f00d626"
server_key_iv: "f02c1a9fcde217c302f7b8ee46c2b7b3"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.159201 20725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:09.181140 20725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:09.185812 20725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:09.186048 20725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:09.205776 20725 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:37481
I20260927 07:12:09.205802 20791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:37481 every 8 connection(s)
I20260927 07:12:09.207237 20725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:09.211191 20792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:09.211905 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20725
I20260927 07:12:09.212116 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:09.212956 30785 external_mini_cluster.cc:1428] Setting key 6b8f79d481e19841ac7aac56a52afc0c
I20260927 07:12:09.222016 20792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.125@KRBTEST.COM" created.
I20260927 07:12:09.248477 20795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.226343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.30.16.126:37481)
0927 07:12:09.227339 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:09.227358 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:09.227970 (+ 612us) server_negotiation.cc:407] Connection header received
0927 07:12:09.230103 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:09.230113 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:09.230433 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:09.230690 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:09.231940 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.233925 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.235545 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.236590 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.236809 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.240654 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:09.240665 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:09.240684 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:09.240726 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:09.244103 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.245012 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.245018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.245020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.245164 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.245704 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.245707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.245709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.246234 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:09.246387 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:09.247076 (+ 689us) server_negotiation.cc:299] Negotiation successful
0927 07:12:09.247383 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":722,"thread_start_us":446,"threads_started":1}
I20260927 07:12:09.248539 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.226343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37481 (local address 127.0.0.1:38906)
0927 07:12:09.227759 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.227806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:09.229679 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:09.230725 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:09.230730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:09.231339 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:09.231700 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.231718 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.234094 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.234098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.235396 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.235408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.236846 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.236851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.236975 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.237716 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:09.237739 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:09.240394 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:09.244239 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.244263 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.244281 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.244875 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.245338 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.245346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.245352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.245592 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.246374 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:09.246386 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:09.246666 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 07:12:09.246949 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 07:12:09.247583 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1134,"mutex_wait_us":66,"spinlock_wait_cycles":65792,"thread_start_us":891,"threads_started":1}
I20260927 07:12:09.251564 20792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:09.252846 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.252305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:42570)
0927 07:12:09.252522 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.252672 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 07:12:09.253152 20741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
I20260927 07:12:09.254926 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.254393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:42572)
0927 07:12:09.254605 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.254703 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 07:12:09.255306 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:46389: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
I20260927 07:12:09.277632 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:46389
--webserver_interface=127.30.16.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:12:09.304702 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } attempt: 1
I20260927 07:12:09.306018 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.305459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52356)
0927 07:12:09.305694 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.305855 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 07:12:09.307057 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.306663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52360)
0927 07:12:09.306827 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.306909 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 07:12:09.307276 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:46389: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
I20260927 07:12:09.375567 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } attempt: 2
I20260927 07:12:09.376857 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.376371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52372)
0927 07:12:09.376617 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.376718 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 07:12:09.378093 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.377710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52380)
0927 07:12:09.377875 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.377952 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:12:09.378388 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:46389: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
I20260927 07:12:09.469620 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } attempt: 3
I20260927 07:12:09.470942 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.470455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52390)
0927 07:12:09.470712 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.470807 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260927 07:12:09.471961 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.471601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52402)
0927 07:12:09.471746 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.471822 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 07:12:09.472129 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:46389: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
I20260927 07:12:09.615373 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } attempt: 4
I20260927 07:12:09.616673 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.616120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52412)
0927 07:12:09.616445 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.616535 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260927 07:12:09.617717 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.617332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52416)
0927 07:12:09.617486 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.617576 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 07:12:09.617893 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:46389: Network error: Client connection negotiation failed: client connection to 127.30.16.125:46389: connect: Connection refused (error 111)
W20260927 07:12:09.632195 20799 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:12:09.632594 20799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:09.632718 20799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:09.632864 20799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:09.640471 20799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:09.640614 20799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:09.640753 20799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:09.640841 20799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:09.640954 20799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:09.651919 20799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:46389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.16.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:09.653445 20799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:09.654935 20799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:09.667532 20804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:09.667573 20805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:09.667618 20807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:09.673745 20799 server_base.cc:1034] running on GCE node
I20260927 07:12:09.674736 20799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:09.676319 20799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:09.677568 20799 hybrid_clock.cc:648] HybridClock initialized: now 1790493129677501 us; error 43 us; skew 500 ppm
I20260927 07:12:09.678072 20799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:09.681855 20799 init.cc:377] Logged in from keytab as kudu/127.30.16.125@KRBTEST.COM (short username kudu)
I20260927 07:12:09.683763 20799 webserver.cc:460] Webserver started at http://127.30.16.125:41225/ using document root <none> and password file <none>
I20260927 07:12:09.684507 20799 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:09.684599 20799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:09.684877 20799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:09.687984 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data/instance:
uuid: "57a4bee088ae4199b5051a032a0263a3"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "b2ad924de3020fe7e99142578c3489de"
server_key_iv: "109f4af63cf8c61010c0577da0362152"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.688877 20799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal/instance:
uuid: "57a4bee088ae4199b5051a032a0263a3"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "b2ad924de3020fe7e99142578c3489de"
server_key_iv: "109f4af63cf8c61010c0577da0362152"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.694356 20799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 07:12:09.698107 20814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:09.699606 20799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:09.699800 20799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "57a4bee088ae4199b5051a032a0263a3"
format_stamp: "Formatted at 2026-09-27 07:12:09 on dist-test-slave-8vqf"
server_key: "b2ad924de3020fe7e99142578c3489de"
server_key_iv: "109f4af63cf8c61010c0577da0362152"
server_key_version: "encryptionkey@0"
I20260927 07:12:09.700053 20799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:09.719309 20799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:09.724303 20799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:09.724712 20799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:09.743780 20799 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.125:46389
I20260927 07:12:09.743803 20865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.125:46389 every 8 connection(s)
I20260927 07:12:09.745270 20799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
I20260927 07:12:09.749280 20866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:09.754597 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20799
I20260927 07:12:09.754796 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-1/wal/instance
I20260927 07:12:09.755282 30785 external_mini_cluster.cc:1428] Setting key 9887b867c92825cdc3bb687da61ea3f4
I20260927 07:12:09.757135 20866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.124@KRBTEST.COM; defaulting to no policy
I20260927 07:12:09.778967 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.760101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.30.16.126:37481)
0927 07:12:09.760629 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:09.760633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:09.761752 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 07:12:09.763984 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:09.763990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:09.764085 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:09.764207 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:09.765845 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.766703 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.768302 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.768738 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.768871 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.772716 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:09.772721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:09.772725 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:09.772764 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:09.775665 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.776480 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.776490 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.776493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.776618 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.777079 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.777085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.777089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.777385 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:09.777554 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:09.778393 (+ 839us) server_negotiation.cc:299] Negotiation successful
0927 07:12:09.778672 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260927 07:12:09.779963 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.760754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37481 (local address 127.0.0.1:47264)
0927 07:12:09.761568 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.761608 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:09.763641 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:09.764469 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:09.764478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:09.765029 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:09.765610 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.765627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.766862 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.766875 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.768140 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.768152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.768977 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.768983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.769230 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.769961 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:09.769984 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:09.772461 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:09.775835 (+ 3374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.775844 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.775863 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.776340 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.776767 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.776772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.776776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.776959 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.777516 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:09.777524 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:09.777951 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 07:12:09.778242 (+ 291us) client_negotiation.cc:240] Negotiation successful
0927 07:12:09.778739 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":315,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.124@KRBTEST.COM" created.
I20260927 07:12:09.782001 20866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:12:09.797214 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.782943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52422)
0927 07:12:09.783128 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.783148 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:09.783248 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:09.784289 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:09.784296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:09.784807 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:09.785025 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.785038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.786427 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.786432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.787349 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.787361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.788258 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.788263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.788496 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.789157 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:09.789174 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:09.791458 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:09.794729 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.794737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.794741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.795101 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.795472 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.795478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.795481 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.795619 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.796202 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:09.796211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:09.796354 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 07:12:09.796509 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 07:12:09.796794 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":23}
I20260927 07:12:09.797276 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.783205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.30.16.125:46389)
0927 07:12:09.783842 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:09.783848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:09.783885 (+ 37us) server_negotiation.cc:407] Connection header received
0927 07:12:09.783927 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:09.783932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:09.784121 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:09.784233 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:09.785203 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.786263 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.787493 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.788044 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.788248 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.791664 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:09.791675 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:09.791689 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:09.791721 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:09.794525 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.795233 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.795242 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.795245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.795367 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.795720 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.795727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.795731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.796082 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:09.796301 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:09.796657 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 07:12:09.796901 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":491,"thread_start_us":356,"threads_started":1}
I20260927 07:12:09.799753 20866 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:09.800928 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.800471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53660)
0927 07:12:09.800674 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.800771 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 07:12:09.801229 20818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:09.802973 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.802504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53668)
0927 07:12:09.802773 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.802850 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260927 07:12:09.803236 20866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
I20260927 07:12:09.814815 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:35883
--webserver_interface=127.30.16.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:41955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:12:09.852527 20866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 1
I20260927 07:12:09.853834 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.853204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53682)
0927 07:12:09.853512 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.853644 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260927 07:12:09.854926 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.854511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53688)
0927 07:12:09.854695 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.854792 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 07:12:09.855141 20866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:09.917119 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } attempt: 5
I20260927 07:12:09.923359 20866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 2
I20260927 07:12:09.924505 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.923973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53694)
0927 07:12:09.924207 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.924319 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
Sep 27 07:12:09 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:12:09.925684 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.925259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53702)
0927 07:12:09.925438 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.925524 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 07:12:09.925969 20866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:09.932286 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.917962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52438)
0927 07:12:09.918190 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.918212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:09.918359 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:09.918769 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:09.918774 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:09.919204 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:09.919385 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.919395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.920359 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.920375 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.921462 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:09.921475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.922259 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.922264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:09.922466 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.923195 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:09.923214 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:09.925896 (+ 2682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:09.929730 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.929734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.929737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.930120 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.930643 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:09.930653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:09.930655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:09.930786 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:09.931537 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:09.931540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:09.931675 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 07:12:09.931809 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 07:12:09.932040 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 07:12:09.932509 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.918013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.30.16.125:46389)
0927 07:12:09.918226 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:09.918234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:09.918280 (+ 46us) server_negotiation.cc:407] Connection header received
0927 07:12:09.918531 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:09.918536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:09.918624 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:09.918734 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:09.919544 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.920225 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.921612 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:09.922071 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:09.922244 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:09.926119 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:09.926126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:09.926130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:09.926170 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:09.929537 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.930264 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.930273 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.930278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.930457 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:09.930908 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:09.930915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:09.930919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:09.931407 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:09.931580 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:09.931960 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 07:12:09.932197 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 07:12:09.933430 20792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:09.934511 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.934099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53706)
0927 07:12:09.934272 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.934377 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 07:12:09.935462 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.935123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53722)
0927 07:12:09.935295 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.935367 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 07:12:09.935689 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:09.986971 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 1
I20260927 07:12:09.988253 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.987777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53730)
0927 07:12:09.987991 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.988089 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 07:12:09.989394 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:09.989047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53732)
0927 07:12:09.989198 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:09.989275 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 07:12:09.989737 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:10.017210 20866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 3
I20260927 07:12:10.018409 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.017955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53734)
0927 07:12:10.018152 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.018252 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 07:12:10.019651 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.019223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53744)
0927 07:12:10.019467 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.019540 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260927 07:12:10.019918 20866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:10.057996 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 2
I20260927 07:12:10.059335 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.058817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53750)
0927 07:12:10.059068 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.059172 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 07:12:10.060364 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.059975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53766)
0927 07:12:10.060135 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.060249 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 07:12:10.060575 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:10.163136 20866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 4
I20260927 07:12:10.164413 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.163868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53770)
0927 07:12:10.164157 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.164259 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260927 07:12:10.165364 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.165043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53778)
0927 07:12:10.165189 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.165266 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 07:12:10.165572 20866 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
I20260927 07:12:10.166779 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 3
W20260927 07:12:10.167423 20874 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260927 07:12:10.167690 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.167278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53790)
0927 07:12:10.167449 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.167536 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 07:12:10.167874 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:10.167991 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:10.168139 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 07:12:10.168749 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.168417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53794)
0927 07:12:10.168582 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.168647 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 07:12:10.169008 20792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:35883: Network error: Client connection negotiation failed: client connection to 127.30.16.124:35883: connect: Connection refused (error 111)
W20260927 07:12:10.176184 20874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:10.176291 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:10.176390 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:10.176476 20874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:10.176589 20874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:10.187543 20874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:41955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:37481,127.30.16.125:46389,127.30.16.124:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:35883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.16.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:10.189045 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:10.190701 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:10.203557 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:10.203660 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:10.203634 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:10.204785 20874 server_base.cc:1034] running on GCE node
I20260927 07:12:10.205888 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:10.207778 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:10.209038 20874 hybrid_clock.cc:648] HybridClock initialized: now 1790493130208964 us; error 35 us; skew 500 ppm
I20260927 07:12:10.209434 20874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493130, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:10.213464 20874 init.cc:377] Logged in from keytab as kudu/127.30.16.124@KRBTEST.COM (short username kudu)
I20260927 07:12:10.215147 20874 webserver.cc:460] Webserver started at http://127.30.16.124:32871/ using document root <none> and password file <none>
I20260927 07:12:10.215871 20874 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:10.216002 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:10.216331 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:10.219532 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data/instance:
uuid: "70e1b92f8fa14dde876cf20532525107"
format_stamp: "Formatted at 2026-09-27 07:12:10 on dist-test-slave-8vqf"
server_key: "548bcf8fe981453cd7ea6b8ad623c72b"
server_key_iv: "12f54d778d57bd32125eb141bb3b20ec"
server_key_version: "encryptionkey@0"
I20260927 07:12:10.220384 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal/instance:
uuid: "70e1b92f8fa14dde876cf20532525107"
format_stamp: "Formatted at 2026-09-27 07:12:10 on dist-test-slave-8vqf"
server_key: "548bcf8fe981453cd7ea6b8ad623c72b"
server_key_iv: "12f54d778d57bd32125eb141bb3b20ec"
server_key_version: "encryptionkey@0"
I20260927 07:12:10.225853 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 07:12:10.230000 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:10.231587 20874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:10.231778 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "70e1b92f8fa14dde876cf20532525107"
format_stamp: "Formatted at 2026-09-27 07:12:10 on dist-test-slave-8vqf"
server_key: "548bcf8fe981453cd7ea6b8ad623c72b"
server_key_iv: "12f54d778d57bd32125eb141bb3b20ec"
server_key_version: "encryptionkey@0"
I20260927 07:12:10.232013 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:10.252966 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:10.257441 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:10.257727 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:10.276796 20874 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.124:35883
I20260927 07:12:10.276816 20940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.124:35883 every 8 connection(s)
I20260927 07:12:10.278210 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
I20260927 07:12:10.281526 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 20874
I20260927 07:12:10.281704 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790492767955610-30785-0/minicluster-data/master-2/wal/instance
I20260927 07:12:10.282258 30785 external_mini_cluster.cc:1428] Setting key 7ea1e5a5c3ab6f16fdc041a0fc09ed01
I20260927 07:12:10.282681 20941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:10.292579 20941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493130, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:12:10.308302 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.284213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.30.16.124:35883)
0927 07:12:10.285004 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.285016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.285059 (+ 43us) server_negotiation.cc:407] Connection header received
0927 07:12:10.287432 (+ 2373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.287456 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.287890 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.288262 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.290106 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.291483 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.293451 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.294338 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.294483 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.299600 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.299610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.299624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.299676 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.303355 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.304254 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.304262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.304272 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.304482 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.304851 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.304859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.304862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.305480 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.305743 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.306224 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.306706 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":361,"threads_started":1}
I20260927 07:12:10.312120 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.295850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:37481 (local address 127.0.0.1:47276)
0927 07:12:10.296677 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.296725 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:10.296867 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:10.297604 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:10.297611 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:10.298496 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:10.298849 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.298861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.300079 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.300083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.301249 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.301261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.302060 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.302066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.302227 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.302856 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:10.302895 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:10.305611 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:10.309363 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.309373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.309378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.309785 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.310197 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.310201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.310205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.310344 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.310968 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:10.310976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:10.311142 (+ 166us) client_negotiation.cc:769] Sending connection context
0927 07:12:10.311317 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 07:12:10.311655 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":417,"threads_started":1}
I20260927 07:12:10.312682 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.295948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47276 (local address 127.30.16.126:37481)
0927 07:12:10.297274 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.297279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.297293 (+ 14us) server_negotiation.cc:407] Connection header received
0927 07:12:10.297342 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.297345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.297409 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.297530 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.299021 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.299912 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.301429 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.301834 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.301973 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.305828 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.305838 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.305847 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.305882 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.309228 (+ 3346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.309948 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.309953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.309957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.310083 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.310454 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.310463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.310467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.310838 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.311005 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.311971 (+ 966us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.312220 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":1105,"threads_started":1}
I20260927 07:12:10.313915 20941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493130, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:12:10.329563 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.314686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:46389 (local address 127.0.0.1:52440)
0927 07:12:10.314876 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.314894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:10.314993 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:10.315277 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:10.315282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:10.315664 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:10.315811 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.315819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.317138 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.317145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.318279 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.318291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.319257 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.319266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.319470 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.320061 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:10.320090 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:10.322974 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:10.327045 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.327050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.327053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.327476 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.327894 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.327899 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.327903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.328095 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.328702 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:10.328707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:10.328891 (+ 184us) client_negotiation.cc:769] Sending connection context
0927 07:12:10.329042 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 07:12:10.329267 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 07:12:10.329963 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.314803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.30.16.125:46389)
0927 07:12:10.314990 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.314995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.315011 (+ 16us) server_negotiation.cc:407] Connection header received
0927 07:12:10.315088 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.315092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.315149 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.315225 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.315967 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.317015 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.318441 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.319068 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.319188 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.323208 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.323214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.323217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.323281 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.326883 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.327611 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.327619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.327626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.327775 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.328209 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.328212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.328213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.328583 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.328731 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.329178 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.329400 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:12:10.330978 20941 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:10.331108 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.311532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47286 (local address 127.30.16.126:37481)
0927 07:12:10.312143 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.312148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.312166 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:12:10.312235 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.312241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.312333 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.312470 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.314041 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.315310 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.316947 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.317574 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.317705 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.323953 (+ 6248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.323958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.323962 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.324031 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.327481 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.328383 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.328389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.328392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.328548 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.328990 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.328993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.328995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.329370 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.329643 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.330475 (+ 832us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.330788 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":272,"threads_started":1}
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493130, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:12:10.346246 20792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 4
I20260927 07:12:10.346487 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.332140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.30.16.124:35883)
0927 07:12:10.332318 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.332324 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.332341 (+ 17us) server_negotiation.cc:407] Connection header received
0927 07:12:10.332389 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.332394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.332459 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.332560 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.333520 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.334714 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.336342 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.337019 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.337154 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.340593 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.340602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.340608 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.340649 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.343830 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.344530 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.344538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.344541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.344663 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.345030 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.345037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.345041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.345333 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.345498 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.345849 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.346092 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":53}
I20260927 07:12:10.346597 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.331863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53802)
0927 07:12:10.332053 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.332081 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:10.332197 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:10.332641 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:10.332648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:10.333164 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:10.333346 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.333357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.334838 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.334842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.336172 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.336192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.337169 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.337173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.337383 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.337994 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:10.338011 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:10.340396 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:10.343969 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.343976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.343980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.344404 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.344784 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.344789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.344793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.344930 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.345455 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:10.345460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:10.345605 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 07:12:10.345772 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 07:12:10.346239 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":172}
I20260927 07:12:10.352423 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.333078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.30.16.125:46389)
0927 07:12:10.333258 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.333265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.333286 (+ 21us) server_negotiation.cc:407] Connection header received
0927 07:12:10.333337 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.333342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.333411 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.333506 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.335153 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.336220 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.337807 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.338284 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.338393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.343740 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.343746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.343750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.343786 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.347713 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.349925 (+ 2212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.349931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.349936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.350113 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.350587 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.350593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.350598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.350986 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.351322 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.351632 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.351883 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 07:12:10.353922 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: Bootstrap starting.
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:10.358438 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:10.358850 20941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:10.360318 20941 log.cc:824] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:10.362352 20796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.346967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53810)
0927 07:12:10.347175 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.347195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:10.347318 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:10.347636 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:10.347640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:10.348178 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:10.348401 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.348412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.349879 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.349883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.351050 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.351093 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.352222 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.352228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.352461 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.353165 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:10.353193 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:10.355709 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:10.359723 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.359729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.359734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.360146 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.360600 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.360604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.360606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.360737 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.361315 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:10.361320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:10.361505 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 07:12:10.361673 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 07:12:10.362018 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 07:12:10.363159 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.347046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.30.16.124:35883)
0927 07:12:10.347208 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.347215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.347238 (+ 23us) server_negotiation.cc:407] Connection header received
0927 07:12:10.347434 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.347439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.347507 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.347607 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.348545 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.349728 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.351269 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.351996 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.352202 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.355921 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.355927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.355934 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.355976 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.359574 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.360283 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.360289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.360293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.360453 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.360850 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.360856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.360860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.361173 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.361355 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.362523 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.362771 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260927 07:12:10.362339 20767 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a5dbc41877a0441a8106567650951b3f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47286
I20260927 07:12:10.367890 20792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: Bootstrap starting.
I20260927 07:12:10.367985 20941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: No bootstrap required, opened a new log
I20260927 07:12:10.371152 20792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:10.372211 20792 log.cc:824] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:10.375819 20792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: No bootstrap required, opened a new log
I20260927 07:12:10.377295 20941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.377696 20941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e1b92f8fa14dde876cf20532525107, State: Initialized, Role: FOLLOWER
I20260927 07:12:10.378170 20941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.379166 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.379557 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.380359 20941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:10.383785 20792 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.384276 20792 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dbc41877a0441a8106567650951b3f, State: Initialized, Role: FOLLOWER
I20260927 07:12:10.384819 20792 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.386180 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.386981 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.387637 20792 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:10.387753 20767 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:12:10.387945 20767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:10.392182 20767 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.397060 20916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5dbc41877a0441a8106567650951b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70e1b92f8fa14dde876cf20532525107"
I20260927 07:12:10.397737 20916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:10.403483 20916 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbc41877a0441a8106567650951b3f in term 1.
I20260927 07:12:10.404225 20767 leader_election.cc:288] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [CANDIDATE]: Term 1 election: Requested vote from peers 57a4bee088ae4199b5051a032a0263a3 (127.30.16.125:46389), 70e1b92f8fa14dde876cf20532525107 (127.30.16.124:35883)
I20260927 07:12:10.405822 20744 leader_election.cc:302] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e1b92f8fa14dde876cf20532525107, a5dbc41877a0441a8106567650951b3f; no voters:
I20260927 07:12:10.388505 20874 master_runner.cc:186] Error getting master registration for 127.30.16.126:37481: OK, instance_id { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" instance_seqno: 1790493129200175 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:12:10.406349 20874 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:12:10.409569 20962 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 07:12:10.409703 20962 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 07:12:10.410022 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:10.415683 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [term 1 LEADER]: Becoming Leader. State: Replica: a5dbc41877a0441a8106567650951b3f, State: Running, Role: LEADER
I20260927 07:12:10.416350 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.424161 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5dbc41877a0441a8106567650951b3f. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.424686 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:10.429554 20973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 07:12:10.439159 20746 tablet.cc:2369] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:12:10.450829 20973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 07:12:10.458643 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.430158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.30.16.125:46389)
0927 07:12:10.430370 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.430376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.431877 (+ 1501us) server_negotiation.cc:407] Connection header received
0927 07:12:10.432340 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.432348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.432433 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.432507 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.437381 (+ 4874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.438515 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.441772 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.442464 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.442593 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.447596 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.447604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.447610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.447665 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.451720 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.455961 (+ 4241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.455968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.455972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.456179 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.456708 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.456713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.456717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.457129 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.457382 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.457958 (+ 576us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.458229 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 07:12:10.464764 20866 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } attempt: 5
I20260927 07:12:10.468518 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.430048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.30.16.126:37481)
0927 07:12:10.430251 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.430260 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.431897 (+ 1637us) server_negotiation.cc:407] Connection header received
0927 07:12:10.432297 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.432306 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.432412 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.432499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.435675 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.436926 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.440500 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.441411 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.441535 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.450293 (+ 8758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.450300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.450305 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.450350 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.454285 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.459046 (+ 4761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.459058 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.459065 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.459459 (+ 394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.461033 (+ 1574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.461040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.461046 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.461490 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.461890 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.464627 (+ 2737us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.465019 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:12:10.472296 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.432100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.30.16.124:35883)
0927 07:12:10.432289 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.432294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.435398 (+ 3104us) server_negotiation.cc:407] Connection header received
0927 07:12:10.435750 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.435756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.435868 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.435952 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.452724 (+ 16772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.453919 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.459791 (+ 5872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.460539 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.460658 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.464399 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.464407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.464412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.464458 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.468600 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.469569 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.469575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.469579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.469770 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.470281 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.470287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.470290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.470686 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.470968 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.471604 (+ 636us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.471860 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:12:10.482609 20916 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [term 1 FOLLOWER]: Refusing update from remote peer a5dbc41877a0441a8106567650951b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:12:10.484762 20961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 27 07:12:10 dist-test-slave-8vqf krb5kdc[20712](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493129, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:10.504912 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5dbc41877a0441a8106567650951b3f. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.505225 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.505620 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.465492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:35883 (local address 127.0.0.1:53836)
0927 07:12:10.466019 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:10.466045 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:10.466198 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:10.472985 (+ 6787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:10.472991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:10.473532 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:10.473772 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.473788 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.476048 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.476055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.477544 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:10.477562 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.478811 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.478820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:10.479192 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.479936 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:10.479963 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:10.490812 (+ 10849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:10.500512 (+ 9700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.500523 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.500531 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.501346 (+ 815us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.502395 (+ 1049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:10.502404 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:10.502410 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:10.502670 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:10.504205 (+ 1535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:10.504214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:10.504485 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 07:12:10.504684 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 07:12:10.505118 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
W20260927 07:12:10.509953 20895 tablet.cc:2369] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:12:10.513007 20984 mvcc.cc:205] Tried to move back new op lower bound from 7333859862418214912 to 7333859862191665152. Current Snapshot: MvccSnapshot[applied={T|T < 7333859862418214912}]
I20260927 07:12:10.519661 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.520155 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:10.521765 20950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.522054 20950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.525616 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.466681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.30.16.124:35883)
0927 07:12:10.472626 (+ 5945us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.472631 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.472649 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:12:10.472695 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.472700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.472774 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.473940 (+ 1166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.474737 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.475851 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.477832 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.478624 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.491066 (+ 12442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.491300 (+ 234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:10.491308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:10.491331 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:10.491368 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:10.500090 (+ 8722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.501681 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.501690 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.501696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.502104 (+ 408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:10.502899 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:10.502954 (+ 55us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:10.502960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:10.503959 (+ 999us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:10.522285 (+ 18326us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.522849 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.523752 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5838,"thread_start_us":22053,"threads_started":1}
I20260927 07:12:10.527174 20973 catalog_manager.cc:1347] Generated new cluster ID: 1f8104912f3d42198552b112712ed7ad
I20260927 07:12:10.527403 20973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:10.553854 20866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: Bootstrap starting.
I20260927 07:12:10.567694 20866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:10.570973 20866 log.cc:824] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:10.604260 20973 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:10.608796 20973 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:10.613085 20866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: No bootstrap required, opened a new log
I20260927 07:12:10.646194 20866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.647420 20866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a4bee088ae4199b5051a032a0263a3, State: Initialized, Role: FOLLOWER
I20260927 07:12:10.648814 20866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } }
I20260927 07:12:10.656577 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.657051 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.658797 20973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f: Generated new TSK 0
I20260927 07:12:10.660060 20973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:10.661203 20841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a5dbc41877a0441a8106567650951b3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "57a4bee088ae4199b5051a032a0263a3"
I20260927 07:12:10.661789 20841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:10.662575 20866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:10.677875 20841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5dbc41877a0441a8106567650951b3f in term 1.
I20260927 07:12:10.678906 20840 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [term 1 FOLLOWER]: Refusing update from remote peer a5dbc41877a0441a8106567650951b3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:12:10.681840 20961 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:12:10.666783 20725 master_runner.cc:186] Error getting master registration for 127.30.16.125:46389: OK, instance_id { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" instance_seqno: 1790493129738453 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:12:10.713428 20725 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:12:10.757725 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5dbc41877a0441a8106567650951b3f. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.758127 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:10.757792 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:10.758266 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 07:12:10.812079 30785 mini_kdc.cc:348] Time spent kdestroy: real 0.112s user 0.000s sys 0.029s
I20260927 07:12:10.854882 21011 catalog_manager.cc:1259] Loaded cluster ID: 1f8104912f3d42198552b112712ed7ad
I20260927 07:12:10.855026 21011 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: loading cluster ID for follower catalog manager: success
I20260927 07:12:10.859709 21011 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: acquiring CA information for follower catalog manager: success
W20260927 07:12:10.862540 20871 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:10.859725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52474 (local address 127.30.16.125:46389)
0927 07:12:10.859982 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.859988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.860583 (+ 595us) server_negotiation.cc:407] Connection header received
0927 07:12:10.861081 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.861087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.861178 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.861265 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.862299 (+ 1034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52474: BlockingRecv error: recv got EOF from 127.0.0.1:52474 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 07:12:10.862962 21011 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 57a4bee088ae4199b5051a032a0263a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 07:12:10.864288 20947 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:10.852997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.30.16.126:37481)
0927 07:12:10.853231 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.853240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.860374 (+ 7134us) server_negotiation.cc:407] Connection header received
0927 07:12:10.860720 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.860727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.860866 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.860959 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.863990 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47298: BlockingRecv error: recv got EOF from 127.0.0.1:47298 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 07:12:10.865195 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:10.860658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.30.16.124:35883)
0927 07:12:10.860859 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.860867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.863605 (+ 2738us) server_negotiation.cc:407] Connection header received
0927 07:12:10.863858 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.863864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.863951 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.864040 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.864959 (+ 919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53838: BlockingRecv error: recv got EOF from 127.0.0.1:53838 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:12:10.944255 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:10.916787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52484 (local address 127.30.16.125:46389)
0927 07:12:10.916999 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.917006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.935300 (+ 18294us) server_negotiation.cc:407] Connection header received
0927 07:12:10.935704 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.935711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.935985 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.936154 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:10.937135 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.938281 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.940462 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:10.941165 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:10.941275 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:10.942460 (+ 1185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:10.943309 (+ 849us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:10.943445 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:10.943751 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 07:12:10.943874 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 07:12:10.990304 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:10.980584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.30.16.124:35883)
0927 07:12:10.980859 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:10.980868 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:10.987906 (+ 7038us) server_negotiation.cc:407] Connection header received
0927 07:12:10.988540 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:10.988549 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:10.988710 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:10.988815 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:10.990016 (+ 1201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53846: BlockingRecv error: recv got EOF from 127.0.0.1:53846 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 07:12:11.038072 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.020652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.30.16.125:46389)
0927 07:12:11.020874 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.020881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.031234 (+ 10353us) server_negotiation.cc:407] Connection header received
0927 07:12:11.031597 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.031605 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.031722 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.031791 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.032502 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.033669 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.035776 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.036451 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.036553 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.037179 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.037338 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.037408 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.037647 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.037740 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 07:12:11.086222 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.076719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53856 (local address 127.30.16.124:35883)
0927 07:12:11.076933 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.076940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.084026 (+ 7086us) server_negotiation.cc:407] Connection header received
0927 07:12:11.084655 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.084662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.084781 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.084862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.085986 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53856: BlockingRecv error: recv got EOF from 127.0.0.1:53856 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:12:11.097862 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5dbc41877a0441a8106567650951b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5dbc41877a0441a8106567650951b3f" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 37481 } } peers { permanent_uuid: "57a4bee088ae4199b5051a032a0263a3" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 46389 } } peers { permanent_uuid: "70e1b92f8fa14dde876cf20532525107" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 35883 } } }
I20260927 07:12:11.098376 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5dbc41877a0441a8106567650951b3f [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:11.148751 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.136683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52492 (local address 127.30.16.125:46389)
0927 07:12:11.136918 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.136924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.142004 (+ 5080us) server_negotiation.cc:407] Connection header received
0927 07:12:11.142217 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.142224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.142330 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.142452 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.143002 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.144208 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.146354 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.147113 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.147264 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.147784 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.147957 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.148064 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.148249 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.148345 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
W20260927 07:12:11.159606 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.156927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.30.16.124:35883)
0927 07:12:11.157154 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.157160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.157723 (+ 563us) server_negotiation.cc:407] Connection header received
0927 07:12:11.158054 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.158066 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.158168 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.158246 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.159355 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53866: BlockingRecv error: recv got EOF from 127.0.0.1:53866 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 07:12:11.186021 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.168258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.30.16.125:46389)
0927 07:12:11.168524 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.168532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.179379 (+ 10847us) server_negotiation.cc:407] Connection header received
0927 07:12:11.179623 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.179631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.179739 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.179855 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.180417 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.181627 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.183925 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.184603 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.184735 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.185262 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.185391 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.185475 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.185644 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.185737 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260927 07:12:11.196594 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.192835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53876 (local address 127.30.16.124:35883)
0927 07:12:11.194096 (+ 1261us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.194104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.195052 (+ 948us) server_negotiation.cc:407] Connection header received
0927 07:12:11.195254 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.195260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.195356 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.195463 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.196357 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53876: BlockingRecv error: recv got EOF from 127.0.0.1:53876 (error 108)
Metrics: {"server-negotiator.queue_time_us":1109}
I20260927 07:12:11.244378 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.236221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.30.16.125:46389)
0927 07:12:11.236420 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.236427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.237015 (+ 588us) server_negotiation.cc:407] Connection header received
0927 07:12:11.237318 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.237324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.237432 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.237558 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.238128 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.239530 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.241880 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.242580 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.242708 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.243365 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.243546 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.243619 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.243905 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.244003 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 07:12:11.254475 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.252005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.30.16.124:35883)
0927 07:12:11.252219 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.252226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.252884 (+ 658us) server_negotiation.cc:407] Connection header received
0927 07:12:11.253074 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.253081 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.253183 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.253303 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.254199 (+ 896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53892: BlockingRecv error: recv got EOF from 127.0.0.1:53892 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 07:12:11.281769 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.274163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52530 (local address 127.30.16.125:46389)
0927 07:12:11.274353 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.274361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.274978 (+ 617us) server_negotiation.cc:407] Connection header received
0927 07:12:11.275228 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.275234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.275337 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.275439 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.276018 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.277253 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.279351 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.280042 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.280162 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.280872 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.281053 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.281122 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.281367 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.281468 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260927 07:12:11.309916 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.307583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.30.16.124:35883)
0927 07:12:11.307795 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.307803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.308352 (+ 549us) server_negotiation.cc:407] Connection header received
0927 07:12:11.308554 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.308561 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.308655 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.308765 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.309651 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53906: BlockingRecv error: recv got EOF from 127.0.0.1:53906 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 07:12:11.351778 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.345634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.30.16.125:46389)
0927 07:12:11.345900 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.345907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.346267 (+ 360us) server_negotiation.cc:407] Connection header received
0927 07:12:11.346440 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.346447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.346603 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.346735 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.347169 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.348320 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.349611 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.350331 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.350478 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.350951 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.351092 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.351198 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.351328 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.351416 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260927 07:12:11.360136 20944 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:11.358151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.30.16.124:35883)
0927 07:12:11.358313 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.358317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.358682 (+ 365us) server_negotiation.cc:407] Connection header received
0927 07:12:11.358869 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.358875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.358965 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.359048 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:11.359880 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53922: BlockingRecv error: recv got EOF from 127.0.0.1:53922 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 07:12:11.411777 20962 catalog_manager.cc:1259] Loaded cluster ID: 1f8104912f3d42198552b112712ed7ad
I20260927 07:12:11.411886 20962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: loading cluster ID for follower catalog manager: success
I20260927 07:12:11.414347 20962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: acquiring CA information for follower catalog manager: success
I20260927 07:12:11.416180 20962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70e1b92f8fa14dde876cf20532525107: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:12:11.433236 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.427868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.30.16.125:46389)
0927 07:12:11.428111 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.428137 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.428726 (+ 589us) server_negotiation.cc:407] Connection header received
0927 07:12:11.428888 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.428892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.428963 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.429044 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.429482 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.430212 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.431679 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.432119 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.432215 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.432655 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.432755 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.432821 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.432956 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.433023 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 07:12:11.445187 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.439274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53936 (local address 127.30.16.124:35883)
0927 07:12:11.439562 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.439568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.439840 (+ 272us) server_negotiation.cc:407] Connection header received
0927 07:12:11.440081 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.440087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.440263 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.440399 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.440765 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.441520 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.442969 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.443391 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.443532 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.444039 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.444606 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.444774 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.444828 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.444912 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 07:12:11.465364 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.451743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.30.16.125:46389)
0927 07:12:11.452028 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.452038 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.458895 (+ 6857us) server_negotiation.cc:407] Connection header received
0927 07:12:11.459210 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.459216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.459409 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.459474 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.460280 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.461055 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.463252 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.463991 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.464132 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.464601 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.464765 (+ 164us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.464864 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.464973 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.465052 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 07:12:11.465713 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.458033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.30.16.124:35883)
0927 07:12:11.458231 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.458237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.459227 (+ 990us) server_negotiation.cc:407] Connection header received
0927 07:12:11.459426 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.459432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.459517 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.459600 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.460302 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.461327 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.463588 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.464288 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.464412 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.464852 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.464990 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.465075 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.465272 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.465363 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 07:12:11.466187 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:11.451650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.30.16.126:37481)
0927 07:12:11.458313 (+ 6663us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:11.458321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:11.458345 (+ 24us) server_negotiation.cc:407] Connection header received
0927 07:12:11.458489 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:11.458495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:11.458783 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:11.459030 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:11.459637 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.460808 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.463136 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:11.464015 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:11.464632 (+ 617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:11.464828 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:11.465309 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:11.465489 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:11.465738 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 07:12:11.465918 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6518,"thread_start_us":6696,"threads_started":1}
I20260927 07:12:11.471344 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20725
I20260927 07:12:11.472590 20787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:11.587507 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20725
I20260927 07:12:11.609262 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20799
I20260927 07:12:11.610364 20861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:11.719699 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20799
I20260927 07:12:11.741196 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 20874
I20260927 07:12:11.742378 20936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:11.850208 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 20874
2026-09-27T07:12:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4488 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 07:12:11 dist-test-slave-8vqf krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 07:12:11 dist-test-slave-8vqf krb5kdc[21114](info): set up 2 sockets
Sep 27 07:12:11 dist-test-slave-8vqf krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260927 07:12:13.031270 30785 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/test-user.keytab.
Sep 27 07:12:13 dist-test-slave-8vqf krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T07:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T07:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.30.16.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
Entry for principal kudu/127.30.16.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab.
I20260927 07:12:13.307230 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39637
--webserver_interface=127.30.16.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 07:12:13.651922 21127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:12:13.652333 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:13.652472 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:13.652612 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:13.660317 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:13.660426 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:13.660534 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:13.660629 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:13.660727 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:13.671010 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.126.keytab
--principal=kudu/127.30.16.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.126:39637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.16.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:13.672454 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:13.673844 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:13.686473 21133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:13.686529 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:13.686529 21135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:13.687978 21127 server_base.cc:1034] running on GCE node
I20260927 07:12:13.688931 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:13.690598 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:13.691898 21127 hybrid_clock.cc:648] HybridClock initialized: now 1790493133691807 us; error 62 us; skew 500 ppm
I20260927 07:12:13.692309 21127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:13 dist-test-slave-8vqf krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:13.695909 21127 init.cc:377] Logged in from keytab as kudu/127.30.16.126@KRBTEST.COM (short username kudu)
I20260927 07:12:13.697587 21127 webserver.cc:460] Webserver started at http://127.30.16.126:44277/ using document root <none> and password file <none>
I20260927 07:12:13.698359 21127 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:13.698499 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:13.698868 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:13.702127 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data/instance:
uuid: "82f267b08f0d46e78165b006fc37ce72"
format_stamp: "Formatted at 2026-09-27 07:12:13 on dist-test-slave-8vqf"
server_key: "a904835ba336f9871ca776ddb0785951"
server_key_iv: "b346ecb3db9ecd91c653c09c15a74a87"
server_key_version: "encryptionkey@0"
I20260927 07:12:13.703058 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance:
uuid: "82f267b08f0d46e78165b006fc37ce72"
format_stamp: "Formatted at 2026-09-27 07:12:13 on dist-test-slave-8vqf"
server_key: "a904835ba336f9871ca776ddb0785951"
server_key_iv: "b346ecb3db9ecd91c653c09c15a74a87"
server_key_version: "encryptionkey@0"
I20260927 07:12:13.708576 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 07:12:13.712332 21142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:13.713877 21127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 07:12:13.714031 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal
uuid: "82f267b08f0d46e78165b006fc37ce72"
format_stamp: "Formatted at 2026-09-27 07:12:13 on dist-test-slave-8vqf"
server_key: "a904835ba336f9871ca776ddb0785951"
server_key_iv: "b346ecb3db9ecd91c653c09c15a74a87"
server_key_version: "encryptionkey@0"
I20260927 07:12:13.714249 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:13.733598 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:13.738209 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:13.738445 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:13.757572 21127 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.126:39637
I20260927 07:12:13.757593 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.126:39637 every 8 connection(s)
I20260927 07:12:13.759232 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/data/info.pb
I20260927 07:12:13.763060 21194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:13.763650 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 21127
I20260927 07:12:13.763856 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-0/wal/instance
I20260927 07:12:13.764364 30785 external_mini_cluster.cc:1428] Setting key 832ea971891cd3ad368d5cf79a52737b
I20260927 07:12:13.769901 21194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:13 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.125@KRBTEST.COM" created.
I20260927 07:12:13.793691 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.775113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.30.16.126:39637)
0927 07:12:13.775823 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:13.775831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:13.775926 (+ 95us) server_negotiation.cc:407] Connection header received
0927 07:12:13.777168 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:13.777173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:13.777409 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:13.777612 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:13.778973 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:13.780140 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:13.781517 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:13.782074 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:13.782268 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:13.786068 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:13.786076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:13.786089 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:13.786122 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:13.789040 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:13.789740 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:13.789750 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:13.789755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:13.789940 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:13.790324 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:13.790332 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:13.790337 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:13.790935 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:13.791154 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:13.791775 (+ 621us) server_negotiation.cc:299] Negotiation successful
0927 07:12:13.792131 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":334,"threads_started":1}
I20260927 07:12:13.793711 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.773753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39637 (local address 127.0.0.1:34068)
0927 07:12:13.774538 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.774651 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:13.776784 (+ 2133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:13.777625 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:13.777631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:13.778349 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:13.778759 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:13.778779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:13.780292 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:13.780297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:13.781367 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:13.781393 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:13.782301 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:13.782307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:13.782517 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:13.783245 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:13.783273 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:13.785827 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:13.789181 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:13.789191 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:13.789201 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:13.789595 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:13.790041 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:13.790044 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:13.790046 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:13.790202 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:13.791093 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:13.791101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:13.791306 (+ 205us) client_negotiation.cc:769] Sending connection context
0927 07:12:13.791596 (+ 290us) client_negotiation.cc:240] Negotiation successful
0927 07:12:13.792277 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":156,"thread_start_us":313,"threads_started":1}
I20260927 07:12:13.797277 21194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:13.798434 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.798016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37814)
0927 07:12:13.798193 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.798314 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 07:12:13.798743 21146 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
I20260927 07:12:13.800328 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.799979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37818)
0927 07:12:13.800128 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.800198 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:12:13.800602 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:40349: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
Entry for principal kudu/127.30.16.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab.
I20260927 07:12:13.824038 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:40349
--webserver_interface=127.30.16.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:12:13.849953 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } attempt: 1
I20260927 07:12:13.851188 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.850719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37822)
0927 07:12:13.850916 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.851047 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 07:12:13.852214 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.851860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37828)
0927 07:12:13.852014 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.852090 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 07:12:13.852421 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:40349: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
I20260927 07:12:13.920707 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } attempt: 2
I20260927 07:12:13.921973 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.921456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37830)
0927 07:12:13.921719 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.921812 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 07:12:13.923094 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:13.922712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37840)
0927 07:12:13.922863 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:13.922964 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 07:12:13.923326 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:40349: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
I20260927 07:12:14.014681 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } attempt: 3
I20260927 07:12:14.016086 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.015542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37854)
0927 07:12:14.015849 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.015950 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 07:12:14.017493 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.017021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37868)
0927 07:12:14.017159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.017285 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 07:12:14.017762 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:40349: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
I20260927 07:12:14.161067 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } attempt: 4
I20260927 07:12:14.162537 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.161931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37870)
0927 07:12:14.162302 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.162397 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260927 07:12:14.163795 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.163368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37876)
0927 07:12:14.163564 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.163654 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 07:12:14.164057 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.125:40349: Network error: Client connection negotiation failed: client connection to 127.30.16.125:40349: connect: Connection refused (error 111)
W20260927 07:12:14.181015 21201 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:12:14.181437 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:14.181568 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:14.181734 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:14.189765 21201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:14.189888 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:14.190029 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:14.190119 21201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:14.190254 21201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:14.201082 21201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.125.keytab
--principal=kudu/127.30.16.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.125:40349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.30.16.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:14.202703 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:14.204257 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:14.216850 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:14.217010 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:14.216980 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:14.218170 21201 server_base.cc:1034] running on GCE node
I20260927 07:12:14.219080 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:14.220602 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:14.221839 21201 hybrid_clock.cc:648] HybridClock initialized: now 1790493134221741 us; error 63 us; skew 500 ppm
I20260927 07:12:14.222195 21201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:14.229395 21201 init.cc:377] Logged in from keytab as kudu/127.30.16.125@KRBTEST.COM (short username kudu)
I20260927 07:12:14.231244 21201 webserver.cc:460] Webserver started at http://127.30.16.125:46589/ using document root <none> and password file <none>
I20260927 07:12:14.232000 21201 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:14.232141 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:14.232481 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:14.235673 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data/instance:
uuid: "65e54441bae44117a9e4b11a0ff21e56"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "9484d22787be3fc1625a81f90410c836"
server_key_iv: "3ddf6ebbf0742d6eb0b9cd3ab2e84549"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.236516 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal/instance:
uuid: "65e54441bae44117a9e4b11a0ff21e56"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "9484d22787be3fc1625a81f90410c836"
server_key_iv: "3ddf6ebbf0742d6eb0b9cd3ab2e84549"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.242270 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 07:12:14.246167 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:14.247866 21201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:14.248073 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal
uuid: "65e54441bae44117a9e4b11a0ff21e56"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "9484d22787be3fc1625a81f90410c836"
server_key_iv: "3ddf6ebbf0742d6eb0b9cd3ab2e84549"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.248318 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:14.279266 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:14.284060 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:14.284371 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:14.304651 21201 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.125:40349
I20260927 07:12:14.304667 21267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.125:40349 every 8 connection(s)
I20260927 07:12:14.306284 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/data/info.pb
I20260927 07:12:14.310185 21268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:14.311640 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 21201
I20260927 07:12:14.311827 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-1/wal/instance
I20260927 07:12:14.312427 30785 external_mini_cluster.cc:1428] Setting key beaef80dad9415eb4870abd32e3ae21c
I20260927 07:12:14.320952 21268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.30.16.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.30.16.124@KRBTEST.COM" created.
I20260927 07:12:14.344640 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.324776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.30.16.126:39637)
0927 07:12:14.325359 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.325364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.326587 (+ 1223us) server_negotiation.cc:407] Connection header received
0927 07:12:14.328616 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.328622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.328720 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.328847 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.330673 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.331476 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.333071 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.333502 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.333639 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.337505 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.337511 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.337514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.337555 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.340543 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.341471 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.341476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.341479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.341680 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.342354 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.342358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.342360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.342761 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.342894 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.343917 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.344119 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":343,"threads_started":1}
I20260927 07:12:14.346153 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.325567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39637 (local address 127.0.0.1:34074)
0927 07:12:14.326381 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.326433 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.328295 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.329131 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.329140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.329783 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.330404 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.330424 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.331668 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.331672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.332911 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.332923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.333757 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.333764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.334025 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.334802 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.334825 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.337214 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.340715 (+ 3501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.340725 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.340744 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.341321 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.341796 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.341801 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.341804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.342050 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.342925 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.342936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.343471 (+ 535us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.343813 (+ 342us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.344459 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":330,"threads_started":1}
I20260927 07:12:14.348376 21268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:12:14.366966 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.349447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37890)
0927 07:12:14.349699 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.349720 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.349839 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.350788 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.350796 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.351379 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.351598 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.351613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.353913 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.353918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.354920 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.354933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.356228 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.356233 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.356393 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.357064 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.357082 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.359334 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.364088 (+ 4754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.364093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.364095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.364486 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.365109 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.365112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.365115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.365253 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.366060 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.366064 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.366178 (+ 114us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.366346 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.366685 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":3}
I20260927 07:12:14.367456 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.349481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.30.16.125:40349)
0927 07:12:14.350250 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.350262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.350287 (+ 25us) server_negotiation.cc:407] Connection header received
0927 07:12:14.350341 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.350347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.350601 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.350760 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.351827 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.353740 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.355063 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.356068 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.356209 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.359572 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.359591 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.359612 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.359655 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.363875 (+ 4220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.364782 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.364795 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.364802 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.364985 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.365369 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.365377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.365383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.365914 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.366138 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.366551 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.366891 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":601,"spinlock_wait_cycles":24320,"thread_start_us":459,"threads_started":1}
I20260927 07:12:14.370117 21268 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:14.371299 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.370853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43940)
0927 07:12:14.371012 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.371146 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260927 07:12:14.371548 21218 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.373203 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.372808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43954)
0927 07:12:14.372940 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.373045 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 07:12:14.373437 21268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
Entry for principal kudu/127.30.16.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab.
I20260927 07:12:14.382951 30785 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
/tmp/dist-test-taskhkCnwT/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:41289
--webserver_interface=127.30.16.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.30.16.84:38991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 07:12:14.422672 21268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 1
I20260927 07:12:14.424002 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.423491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43958)
0927 07:12:14.423725 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.423848 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 07:12:14.425113 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.424723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43970)
0927 07:12:14.424932 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.425008 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 07:12:14.425302 21268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.463341 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } attempt: 5
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
I20260927 07:12:14.477352 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.464230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37892)
0927 07:12:14.464494 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.464515 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.464702 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.465085 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.465090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.465527 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.465715 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.465725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.466854 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.466863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.467749 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.467760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.468549 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.468555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.468777 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.469445 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.469463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.471624 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.474815 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.474821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.474826 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.475224 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.475615 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.475621 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.475624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.475772 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.476317 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.476330 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.476497 (+ 167us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.476646 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.476977 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260927 07:12:14.477360 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.464271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.30.16.125:40349)
0927 07:12:14.464507 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.464515 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.464619 (+ 104us) server_negotiation.cc:407] Connection header received
0927 07:12:14.464860 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.464864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.464933 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.465046 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.465938 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.466718 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.467906 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.468377 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.468518 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.471844 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.471849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.471853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.471888 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.474674 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.475359 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.475365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.475369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.475498 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.475888 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.475895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.475898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.476195 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.476368 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.476741 (+ 373us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.476977 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260927 07:12:14.478699 21194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:14.479843 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.479441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43976)
0927 07:12:14.479610 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.479688 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 07:12:14.480784 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.480443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:43990)
0927 07:12:14.480588 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.480679 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 07:12:14.481011 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.493516 21268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 2
I20260927 07:12:14.494732 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.494199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44002)
0927 07:12:14.494392 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.494497 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 07:12:14.495780 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.495449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44014)
0927 07:12:14.495605 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.495671 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 07:12:14.496039 21268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.532332 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 1
I20260927 07:12:14.533854 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.533097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44022)
0927 07:12:14.533411 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.533606 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 07:12:14.535473 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.534945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44030)
0927 07:12:14.535204 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.535312 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 07:12:14.535874 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.587246 21268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 3
I20260927 07:12:14.588409 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.587934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44032)
0927 07:12:14.588133 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.588232 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 07:12:14.589560 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.589141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44048)
0927 07:12:14.589356 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.589446 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260927 07:12:14.589756 21268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.604157 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 2
I20260927 07:12:14.605625 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.604948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44056)
0927 07:12:14.605250 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.605382 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 07:12:14.606964 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.606499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44060)
0927 07:12:14.606697 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.606799 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260927 07:12:14.607254 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.713558 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 3
I20260927 07:12:14.715018 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.714357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44066)
0927 07:12:14.714666 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.714811 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 07:12:14.716230 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.715829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44070)
0927 07:12:14.716041 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.716118 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260927 07:12:14.716529 21194 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
I20260927 07:12:14.732981 21268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 4
I20260927 07:12:14.734341 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.733720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44078)
0927 07:12:14.734019 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.734131 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 07:12:14.735534 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.735147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44086)
0927 07:12:14.735313 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.735398 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 07:12:14.735826 21268 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.30.16.124:41289: Network error: Client connection negotiation failed: client connection to 127.30.16.124:41289: connect: Connection refused (error 111)
W20260927 07:12:14.742079 21276 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 07:12:14.742470 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 07:12:14.742630 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 07:12:14.742762 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 07:12:14.750515 21276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 07:12:14.750681 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 07:12:14.750795 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 07:12:14.750890 21276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 07:12:14.750974 21276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 07:12:14.761525 21276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.16.84:38991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.30.16.126:39637,127.30.16.125:40349,127.30.16.124:41289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/krb5kdc/kudu_127.30.16.124.keytab
--principal=kudu/127.30.16.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.16.124:41289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.30.16.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 05:59:20 UTC on ccycloud.quasar-zavphu.root.comops.site
build id 4675737
ASAN enabled
I20260927 07:12:14.762897 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 07:12:14.764298 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 07:12:14.776410 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:14.776381 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 07:12:14.776551 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 07:12:14.777434 21276 server_base.cc:1034] running on GCE node
I20260927 07:12:14.778328 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 07:12:14.779906 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 07:12:14.781167 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790493134781094 us; error 49 us; skew 500 ppm
I20260927 07:12:14.781551 21276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 07:12:14.784983 21276 init.cc:377] Logged in from keytab as kudu/127.30.16.124@KRBTEST.COM (short username kudu)
I20260927 07:12:14.786517 21276 webserver.cc:460] Webserver started at http://127.30.16.124:43907/ using document root <none> and password file <none>
I20260927 07:12:14.787300 21276 fs_manager.cc:359] Metadata directory not provided
I20260927 07:12:14.787443 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 07:12:14.787770 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 07:12:14.791018 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data/instance:
uuid: "65b7d92e625344ed8625131386dcd5b5"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "3338224ef23842657e826fac7b199221"
server_key_iv: "2f4962749cf0784122caee8c0b0ece8b"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.791935 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal/instance:
uuid: "65b7d92e625344ed8625131386dcd5b5"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "3338224ef23842657e826fac7b199221"
server_key_iv: "2f4962749cf0784122caee8c0b0ece8b"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.797521 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 07:12:14.801312 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 07:12:14.802884 21276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 07:12:14.803112 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data,/tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal
uuid: "65b7d92e625344ed8625131386dcd5b5"
format_stamp: "Formatted at 2026-09-27 07:12:14 on dist-test-slave-8vqf"
server_key: "3338224ef23842657e826fac7b199221"
server_key_iv: "2f4962749cf0784122caee8c0b0ece8b"
server_key_version: "encryptionkey@0"
I20260927 07:12:14.803377 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 07:12:14.822806 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 07:12:14.827702 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 07:12:14.828011 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 07:12:14.849377 21276 rpc_server.cc:307] RPC server started. Bound to: 127.30.16.124:41289
I20260927 07:12:14.849396 21342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.30.16.124:41289 every 8 connection(s)
I20260927 07:12:14.850921 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/data/info.pb
I20260927 07:12:14.855023 21343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 07:12:14.860500 30785 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu as pid 21276
I20260927 07:12:14.860682 30785 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhkCnwT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790492767955610-30785-0/minicluster-data/master-2/wal/instance
I20260927 07:12:14.861191 30785 external_mini_cluster.cc:1428] Setting key 19120864d812684f54a845865133b80b
I20260927 07:12:14.862116 21343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:14.882066 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.865140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.30.16.126:39637)
0927 07:12:14.865637 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.865646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.866174 (+ 528us) server_negotiation.cc:407] Connection header received
0927 07:12:14.866747 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.866752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.866851 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.866978 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.868241 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.869003 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.870505 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.870993 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.871124 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.874832 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.874837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.874841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.874879 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.878306 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.879210 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.879215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.879219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.879389 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.879938 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.879941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.879944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.880268 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.880435 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.881374 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.881644 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":243,"threads_started":1}
I20260927 07:12:14.883491 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.865190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.126:39637 (local address 127.0.0.1:34088)
0927 07:12:14.866022 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.866059 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.866464 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.867038 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.867045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.867711 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.868078 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.868095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.869160 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.869172 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.870346 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.870358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.871247 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.871255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.871459 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.872197 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.872231 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.874596 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.878456 (+ 3860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.878465 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.878475 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.879083 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.879505 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.879509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.879511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.879808 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.880420 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.880427 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.880924 (+ 497us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.881195 (+ 271us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.881705 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":487,"threads_started":1}
I20260927 07:12:14.883529 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.863164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44098 (local address 127.30.16.124:41289)
0927 07:12:14.863887 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.863899 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.863940 (+ 41us) server_negotiation.cc:407] Connection header received
0927 07:12:14.866459 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.866480 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.866782 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.866990 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.868428 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.869614 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.870909 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.871448 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.871588 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.875762 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.875774 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.875792 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.875830 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.879477 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.880309 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.880319 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.880324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.880504 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.880922 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.880925 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.880927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.881385 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.881686 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.882045 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.882660 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":307,"threads_started":1}
I20260927 07:12:14.885105 21343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } has no permanent_uuid. Determining permanent_uuid...
I20260927 07:12:14.893779 21194 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 4
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.126@KRBTEST.COM
I20260927 07:12:14.901386 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.885938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.125:40349 (local address 127.0.0.1:37906)
0927 07:12:14.886111 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.886130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.886256 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.886832 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.886837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.887405 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.887615 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.887627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.889089 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.889092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.890448 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.890462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.891492 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.891496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.891647 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.892268 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.892286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.894837 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.898667 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.898673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.898677 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.899235 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.899703 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.899711 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.899725 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.899920 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.900471 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.900476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.900661 (+ 185us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.900875 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.901131 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 07:12:14.901615 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.886284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37906 (local address 127.30.16.125:40349)
0927 07:12:14.886482 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.886487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.886502 (+ 15us) server_negotiation.cc:407] Connection header received
0927 07:12:14.886614 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.886617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.886678 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.886798 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.887783 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.888928 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.890649 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.891318 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.891485 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.895065 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.895072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.895077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.895120 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.898523 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.899365 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.899370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.899382 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.899565 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.900052 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.900063 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.900066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.900332 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.900494 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.900986 (+ 492us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.901214 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 07:12:14.902645 21343 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.126@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:14.905139 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.887132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.30.16.126:39637)
0927 07:12:14.887308 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.887314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.887332 (+ 18us) server_negotiation.cc:407] Connection header received
0927 07:12:14.887430 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.887435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.887518 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.887622 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.889391 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.890434 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.892144 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.892637 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.892793 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.898095 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.898105 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.898112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.898156 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.901853 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.902898 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.902902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.902904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.903049 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.903529 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.903533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.903537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.903933 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.904352 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.904543 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.904788 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:12:14.915550 21346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.894508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.30.16.124:41289)
0927 07:12:14.894703 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.894709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.894806 (+ 97us) server_negotiation.cc:407] Connection header received
0927 07:12:14.895030 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.895037 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.895110 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.895208 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.896195 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.897325 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.898987 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.899591 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.899787 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.906506 (+ 6719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.906516 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.906524 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.906607 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.912434 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.913351 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.913357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.913362 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.913501 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.913983 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.913990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.913996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.914310 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.914488 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.914943 (+ 455us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.915213 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 07:12:14.915570 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.894407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44100)
0927 07:12:14.894706 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.894726 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.894879 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.895274 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.895280 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.895808 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.896021 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.896036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.897487 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.897492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.898812 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.898828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.899825 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.899833 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.900368 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.902100 (+ 1732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.902116 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.906198 (+ 4082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.912575 (+ 6377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.912581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.912586 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.913191 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.913635 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.913641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.913644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.913836 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.914447 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.914452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.914677 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.914826 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.915150 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.30.16.125@KRBTEST.COM
Sep 27 07:12:14 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.124@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:14.922443 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: Bootstrap starting.
I20260927 07:12:14.925272 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.906894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37922 (local address 127.30.16.125:40349)
0927 07:12:14.907054 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.907059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.907170 (+ 111us) server_negotiation.cc:407] Connection header received
0927 07:12:14.907363 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.907369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.907444 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.907551 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.909146 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.910632 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.912376 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.913068 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.913206 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.918620 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.918629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.918637 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.918733 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.922110 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.923081 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.923088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.923091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.923253 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.923661 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.923666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.923670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.923985 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.924327 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.924621 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.924882 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 07:12:14.927140 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.903384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44104)
0927 07:12:14.903554 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:14.903573 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:14.903671 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:14.911561 (+ 7890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:14.911569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:14.912197 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:14.912392 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.912404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.913729 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.913734 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.915263 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:14.915280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.916479 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.916487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:14.916722 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.917449 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:14.917471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:14.920233 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:14.924202 (+ 3969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.924210 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.924216 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.924756 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.925176 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:14.925180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:14.925183 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:14.925319 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:14.926056 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:14.926060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:14.926221 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 07:12:14.926376 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 07:12:14.926710 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 07:12:14.927156 21194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:14.928637 21194 log.cc:824] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:14.928632 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.903467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44104 (local address 127.30.16.124:41289)
0927 07:12:14.911202 (+ 7735us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.911208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.911236 (+ 28us) server_negotiation.cc:407] Connection header received
0927 07:12:14.911291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.911298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.911379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.911500 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.912576 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.913574 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.915433 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.916167 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.916285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.920479 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.920486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.920493 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.920530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:14.924042 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.924909 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.924916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.924922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.925059 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:14.925432 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:14.925436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:14.925439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:14.925835 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:14.926002 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:14.927978 (+ 1976us) server_negotiation.cc:299] Negotiation successful
0927 07:12:14.928204 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7584,"thread_start_us":7317,"threads_started":1}
I20260927 07:12:14.931075 30785 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 07:12:14.933326 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: Bootstrap starting.
I20260927 07:12:14.933527 21169 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "82f267b08f0d46e78165b006fc37ce72"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34096
I20260927 07:12:14.936618 21343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:14.938021 21343 log.cc:824] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:14.938328 21194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: No bootstrap required, opened a new log
I20260927 07:12:14.940824 21343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: No bootstrap required, opened a new log
I20260927 07:12:14.944482 21194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:14.944866 21194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f267b08f0d46e78165b006fc37ce72, State: Initialized, Role: FOLLOWER
I20260927 07:12:14.945406 21194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:14.946426 21352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:14.946842 21352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:14.947688 21194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:14.947829 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 07:12:14.948084 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:14.951881 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:14.952157 21343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:14.952634 21343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b7d92e625344ed8625131386dcd5b5, State: Initialized, Role: FOLLOWER
I20260927 07:12:14.953186 21343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:14.954972 21169 leader_election.cc:288] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [CANDIDATE]: Term 1 election: Requested vote from peers 65e54441bae44117a9e4b11a0ff21e56 (127.30.16.125:40349), 65b7d92e625344ed8625131386dcd5b5 (127.30.16.124:41289)
I20260927 07:12:14.955765 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:14.956202 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:14.957196 21343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:14.957401 21318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82f267b08f0d46e78165b006fc37ce72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65b7d92e625344ed8625131386dcd5b5"
I20260927 07:12:14.957934 21318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:14.963150 21318 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f267b08f0d46e78165b006fc37ce72 in term 1.
I20260927 07:12:14.964048 21144 leader_election.cc:302] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65b7d92e625344ed8625131386dcd5b5, 82f267b08f0d46e78165b006fc37ce72; no voters:
I20260927 07:12:14.964659 21352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 07:12:14.980186 21352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [term 1 LEADER]: Becoming Leader. State: Replica: 82f267b08f0d46e78165b006fc37ce72, State: Running, Role: LEADER
I20260927 07:12:14.980937 21352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
W20260927 07:12:14.991374 21148 tablet.cc:2369] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:12:15.011895 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.961749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37932 (local address 127.30.16.125:40349)
0927 07:12:14.962005 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.962013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.966482 (+ 4469us) server_negotiation.cc:407] Connection header received
0927 07:12:14.971916 (+ 5434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.971925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.972043 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.972134 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.981702 (+ 9568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.983019 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.985675 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.986674 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.986813 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:14.995935 (+ 9122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:14.995946 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:14.995954 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:14.996020 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:15.001203 (+ 5183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.005795 (+ 4592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.005806 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.005813 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.006396 (+ 583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.007634 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.007643 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.007650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.008585 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:15.009109 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.010434 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.010996 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 07:12:15.015471 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f267b08f0d46e78165b006fc37ce72. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.016345 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:15.016526 21379 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:12:15.022496 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.966779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.30.16.124:41289)
0927 07:12:14.967004 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.967013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.983672 (+ 16659us) server_negotiation.cc:407] Connection header received
0927 07:12:14.984200 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.984210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.984358 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.984450 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.988888 (+ 4438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.990221 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.994397 (+ 4176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.995372 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.995527 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.001647 (+ 6120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:15.001657 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:15.001666 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:15.001726 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:15.010753 (+ 9027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.016482 (+ 5729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.016491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.016498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.016718 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.017591 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.017603 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.017610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.019712 (+ 2102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:15.020066 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.021589 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.021961 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:12:15.025347 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:14.961034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.30.16.126:39637)
0927 07:12:14.961242 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:14.961250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:14.964663 (+ 3413us) server_negotiation.cc:407] Connection header received
0927 07:12:14.966803 (+ 2140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:14.966812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:14.966929 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:14.967023 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:14.977468 (+ 10445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.979094 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.981827 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:14.982516 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:14.982658 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.002824 (+ 20166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:15.002836 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:15.002844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:15.002934 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:15.008233 (+ 5299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.012416 (+ 4183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.012427 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.012434 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.012713 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.015864 (+ 3151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.015884 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.015891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.017767 (+ 1876us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:15.021974 (+ 4207us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.022650 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.023818 (+ 1168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 07:12:15.025700 21382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 07:12:15.035050 21268 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } attempt: 5
I20260927 07:12:15.023480 21127 master_runner.cc:186] Error getting master registration for 127.30.16.125:40349: OK, instance_id { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" instance_seqno: 1790493134298771 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:12:15.036618 21127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 07:12:15.036931 21382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 07:12:15 dist-test-slave-8vqf krb5kdc[21114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790493134, etypes {rep=17 tkt=17 ses=17}, kudu/127.30.16.125@KRBTEST.COM for kudu/127.30.16.124@KRBTEST.COM
I20260927 07:12:15.051725 21386 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 07:12:15.059268 21318 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f267b08f0d46e78165b006fc37ce72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:12:15.060195 21371 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:12:15.070854 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.035759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.30.16.124:41289 (local address 127.0.0.1:44130)
0927 07:12:15.036169 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 07:12:15.036190 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 07:12:15.036321 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 07:12:15.037430 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 07:12:15.037435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 07:12:15.037993 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 07:12:15.038213 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:15.038226 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.039727 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.039733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:15.041136 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 07:12:15.041152 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.042760 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.042769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 07:12:15.043142 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.044054 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 07:12:15.044089 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 07:12:15.047945 (+ 3856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 07:12:15.067704 (+ 19759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:15.067710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:15.067714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:15.068354 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:15.068872 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 07:12:15.068883 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 07:12:15.068887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 07:12:15.069063 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 07:12:15.069749 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 07:12:15.069757 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 07:12:15.069940 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 07:12:15.070103 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 07:12:15.070441 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260927 07:12:15.070859 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.036664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.30.16.124:41289)
0927 07:12:15.036943 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.036952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.036974 (+ 22us) server_negotiation.cc:407] Connection header received
0927 07:12:15.037040 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.037045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.037125 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.037226 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.038415 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.039554 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.041349 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.042262 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.042429 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.048185 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 07:12:15.048192 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 07:12:15.048198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 07:12:15.048244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 07:12:15.067484 (+ 19240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.068557 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.068564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.068570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.068740 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 07:12:15.069197 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 07:12:15.069204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 07:12:15.069208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 07:12:15.069626 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 07:12:15.069851 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.070205 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.070442 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260927 07:12:15.076850 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: Bootstrap starting.
I20260927 07:12:15.059796 21276 master_runner.cc:186] Error getting master registration for 127.30.16.125:40349: OK, instance_id { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" instance_seqno: 1790493134298771 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 07:12:15.078732 21276 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 07:12:15.080197 21297 tablet.cc:2369] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 07:12:15.081185 21268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: Neither blocks nor log segments found. Creating new log.
I20260927 07:12:15.082659 21268 log.cc:824] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: Log is configured to *not* fsync() on all Append() calls
I20260927 07:12:15.087467 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f267b08f0d46e78165b006fc37ce72. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.087826 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:15.092710 21268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: No bootstrap required, opened a new log
I20260927 07:12:15.093917 21388 mvcc.cc:205] Tried to move back new op lower bound from 7333859881191370752 to 7333859880904577024. Current Snapshot: MvccSnapshot[applied={T|T < 7333859881191370752}]
I20260927 07:12:15.100172 21359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.100468 21359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:15.103384 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.103668 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:15.104132 21382 catalog_manager.cc:1347] Generated new cluster ID: a38d235a3e9547ce8a2c1d437759d9bf
I20260927 07:12:15.104233 21382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 07:12:15.111112 21268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:15.111672 21268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e54441bae44117a9e4b11a0ff21e56, State: Initialized, Role: FOLLOWER
I20260927 07:12:15.112294 21268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } }
I20260927 07:12:15.119385 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.123417 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:15.123286 21268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 07:12:15.124017 21242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 07:12:15.126081 21382 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 07:12:15.124262 21243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "82f267b08f0d46e78165b006fc37ce72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "65e54441bae44117a9e4b11a0ff21e56"
I20260927 07:12:15.128000 21382 catalog_manager.cc:1504] Loading token signing keys...
I20260927 07:12:15.128784 21242 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [term 1 FOLLOWER]: Refusing update from remote peer 82f267b08f0d46e78165b006fc37ce72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 07:12:15.132166 21243 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f267b08f0d46e78165b006fc37ce72 in term 1.
I20260927 07:12:15.137007 21352 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 07:12:15.150002 21382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72: Generated new TSK 0
I20260927 07:12:15.151453 21382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 07:12:15.175271 21400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.175722 21400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:15.176226 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82f267b08f0d46e78165b006fc37ce72. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.176519 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 07:12:15.178026 21371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82f267b08f0d46e78165b006fc37ce72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f267b08f0d46e78165b006fc37ce72" member_type: VOTER last_known_addr { host: "127.30.16.126" port: 39637 } } peers { permanent_uuid: "65e54441bae44117a9e4b11a0ff21e56" member_type: VOTER last_known_addr { host: "127.30.16.125" port: 40349 } } peers { permanent_uuid: "65b7d92e625344ed8625131386dcd5b5" member_type: VOTER last_known_addr { host: "127.30.16.124" port: 41289 } } }
I20260927 07:12:15.178334 21371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82f267b08f0d46e78165b006fc37ce72 [sys.catalog]: This master's current role is: LEADER
I20260927 07:12:15.198145 21409 catalog_manager.cc:1259] Loaded cluster ID: a38d235a3e9547ce8a2c1d437759d9bf
I20260927 07:12:15.198365 21409 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: loading cluster ID for follower catalog manager: success
I20260927 07:12:15.202879 21409 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: acquiring CA information for follower catalog manager: success
I20260927 07:12:15.205968 21409 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65e54441bae44117a9e4b11a0ff21e56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 07:12:15.253401 21347 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.234458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.30.16.126:39637)
0927 07:12:15.234663 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.234672 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.251217 (+ 16545us) server_negotiation.cc:407] Connection header received
0927 07:12:15.251578 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.251587 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.251706 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.251790 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.253124 (+ 1334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34112: BlockingRecv error: recv got EOF from 127.0.0.1:34112 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 07:12:15.253755 21273 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.234110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.30.16.125:40349)
0927 07:12:15.234343 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.234350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.251237 (+ 16887us) server_negotiation.cc:407] Connection header received
0927 07:12:15.251485 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.251496 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.251597 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.251705 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.253510 (+ 1805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37944: BlockingRecv error: recv got EOF from 127.0.0.1:37944 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 07:12:15.256019 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.251977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.30.16.124:41289)
0927 07:12:15.252248 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.252257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.254112 (+ 1855us) server_negotiation.cc:407] Connection header received
0927 07:12:15.254230 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.254239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.254369 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.254474 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.255673 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44138: BlockingRecv error: recv got EOF from 127.0.0.1:44138 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 07:12:15.269414 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.261224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.30.16.125:40349)
0927 07:12:15.261445 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.261459 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.263516 (+ 2057us) server_negotiation.cc:407] Connection header received
0927 07:12:15.263774 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.263779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.263934 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.264097 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.264528 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.265339 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.266964 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.267559 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.267687 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.268189 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.268727 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.268899 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.268953 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.269042 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 07:12:15.285046 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.280201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.30.16.124:41289)
0927 07:12:15.280396 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.280401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.283517 (+ 3116us) server_negotiation.cc:407] Connection header received
0927 07:12:15.283790 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.283795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.283907 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.284048 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.284843 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44152: BlockingRecv error: recv got EOF from 127.0.0.1:44152 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 07:12:15.300009 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.293353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37952 (local address 127.30.16.125:40349)
0927 07:12:15.293640 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.293650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.295156 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 07:12:15.295395 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.295400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.295467 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.295568 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.295995 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.296740 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.298102 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.298608 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.298734 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.299144 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.299280 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.299378 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.299580 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.299665 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 07:12:15.348718 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.345074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.30.16.124:41289)
0927 07:12:15.345318 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.345326 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.347186 (+ 1860us) server_negotiation.cc:407] Connection header received
0927 07:12:15.347424 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.347437 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.347516 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.347631 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.348455 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44154: BlockingRecv error: recv got EOF from 127.0.0.1:44154 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 07:12:15.360199 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.354441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37968 (local address 127.30.16.125:40349)
0927 07:12:15.354653 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.354660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.355057 (+ 397us) server_negotiation.cc:407] Connection header received
0927 07:12:15.355286 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.355298 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.355372 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.355478 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.355942 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.356730 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.358280 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.358846 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.358985 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.359410 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.359540 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.359639 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.359788 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.359898 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 07:12:15.372763 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.367096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.30.16.124:41289)
0927 07:12:15.367322 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.367331 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.371201 (+ 3870us) server_negotiation.cc:407] Connection header received
0927 07:12:15.371426 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.371432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.371523 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.371650 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.372523 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44164: BlockingRecv error: recv got EOF from 127.0.0.1:44164 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 07:12:15.385618 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.379648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.30.16.125:40349)
0927 07:12:15.379915 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.379925 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.380458 (+ 533us) server_negotiation.cc:407] Connection header received
0927 07:12:15.380683 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.380689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.380794 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.380895 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.381338 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.382331 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.383876 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.384339 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.384490 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.384879 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.384988 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.385087 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.385210 (+ 123us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.385293 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260927 07:12:15.394019 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.391688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44172 (local address 127.30.16.124:41289)
0927 07:12:15.392022 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.392031 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.392576 (+ 545us) server_negotiation.cc:407] Connection header received
0927 07:12:15.392793 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.392799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.392891 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.393001 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.393793 (+ 792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44172: BlockingRecv error: recv got EOF from 127.0.0.1:44172 (error 108)
Metrics: {"server-negotiator.queue_time_us":174}
I20260927 07:12:15.411778 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.405382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.30.16.125:40349)
0927 07:12:15.405606 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.405615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.406330 (+ 715us) server_negotiation.cc:407] Connection header received
0927 07:12:15.406589 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.406594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.406680 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.406785 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.407313 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.408390 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.409859 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.410353 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.410499 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.410970 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.411082 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.411188 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.411335 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.411426 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 07:12:15.424592 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.418480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.30.16.124:41289)
0927 07:12:15.418683 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.418692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.423111 (+ 4419us) server_negotiation.cc:407] Connection header received
0927 07:12:15.423339 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.423345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.423436 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.423546 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.424326 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44188: BlockingRecv error: recv got EOF from 127.0.0.1:44188 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 07:12:15.450115 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.444353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.30.16.125:40349)
0927 07:12:15.444599 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.444608 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.445133 (+ 525us) server_negotiation.cc:407] Connection header received
0927 07:12:15.445359 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.445364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.445439 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.445539 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.445957 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.446849 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.448257 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.448703 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.448840 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.449334 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.449479 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.449580 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.449732 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.449818 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260927 07:12:15.459054 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.456664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.30.16.124:41289)
0927 07:12:15.456898 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.456908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.457508 (+ 600us) server_negotiation.cc:407] Connection header received
0927 07:12:15.457772 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.457777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.457851 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.457965 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.458813 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44196: BlockingRecv error: recv got EOF from 127.0.0.1:44196 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 07:12:15.501737 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.495652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.30.16.125:40349)
0927 07:12:15.495928 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.495937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.496538 (+ 601us) server_negotiation.cc:407] Connection header received
0927 07:12:15.496774 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.496778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.496852 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.496965 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.497437 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.498191 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.499848 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.500370 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.500517 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.500929 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.501036 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.501136 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.501289 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.501374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 07:12:15.511077 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.508982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44206 (local address 127.30.16.124:41289)
0927 07:12:15.509152 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.509158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.509533 (+ 375us) server_negotiation.cc:407] Connection header received
0927 07:12:15.509779 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.509786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.509889 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.510004 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.510831 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44206: BlockingRecv error: recv got EOF from 127.0.0.1:44206 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 07:12:15.588294 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.582493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.30.16.125:40349)
0927 07:12:15.582754 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.582763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.583324 (+ 561us) server_negotiation.cc:407] Connection header received
0927 07:12:15.583564 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.583570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.583656 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.583759 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.584214 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.585003 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.586321 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.586860 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.587019 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.587437 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.587556 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.587659 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.587845 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.587962 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 07:12:15.599128 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.596819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.30.16.124:41289)
0927 07:12:15.597079 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.597089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.597658 (+ 569us) server_negotiation.cc:407] Connection header received
0927 07:12:15.597877 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.597883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.597973 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.598087 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.598889 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44218: BlockingRecv error: recv got EOF from 127.0.0.1:44218 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 07:12:15.746054 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:15.740437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.30.16.125:40349)
0927 07:12:15.740756 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.740763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.740939 (+ 176us) server_negotiation.cc:407] Connection header received
0927 07:12:15.741103 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.741108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.741192 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.741293 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:15.741743 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.742617 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.744159 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:15.744762 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:15.744874 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:15.745266 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:15.745419 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:15.745492 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:15.745621 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 07:12:15.745693 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260927 07:12:15.754679 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:15.752609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44226 (local address 127.30.16.124:41289)
0927 07:12:15.752856 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:15.752862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:15.753209 (+ 347us) server_negotiation.cc:407] Connection header received
0927 07:12:15.753353 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:15.753357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:15.753434 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:15.753574 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:15.754408 (+ 834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44226: BlockingRecv error: recv got EOF from 127.0.0.1:44226 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 07:12:16.020411 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.015094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.30.16.125:40349)
0927 07:12:16.015416 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.015423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.015551 (+ 128us) server_negotiation.cc:407] Connection header received
0927 07:12:16.015792 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.015801 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.015887 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.015994 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.016463 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.017209 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.018715 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.019133 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.019307 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.019696 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.019803 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.019904 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.020057 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.020139 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260927 07:12:16.029079 21349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 07:12:16.027080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.30.16.124:41289)
0927 07:12:16.027310 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.027317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.027596 (+ 279us) server_negotiation.cc:407] Connection header received
0927 07:12:16.027867 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.027872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.027962 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.028076 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 07:12:16.028889 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44230: BlockingRecv error: recv got EOF from 127.0.0.1:44230 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 07:12:16.056417 21386 catalog_manager.cc:1259] Loaded cluster ID: a38d235a3e9547ce8a2c1d437759d9bf
I20260927 07:12:16.056602 21386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: loading cluster ID for follower catalog manager: success
I20260927 07:12:16.059526 21386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: acquiring CA information for follower catalog manager: success
I20260927 07:12:16.061551 21386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65b7d92e625344ed8625131386dcd5b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 07:12:16.554960 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.546406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38026 (local address 127.30.16.125:40349)
0927 07:12:16.546965 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.546971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.546998 (+ 27us) server_negotiation.cc:407] Connection header received
0927 07:12:16.547301 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.547309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.547654 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.547763 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.548508 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.549986 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.551939 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.552772 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.552913 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.553540 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.553679 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.553762 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.554444 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.554533 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":303,"threads_started":1}
I20260927 07:12:16.585398 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.563030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44232 (local address 127.30.16.124:41289)
0927 07:12:16.578183 (+ 15153us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.578188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.578214 (+ 26us) server_negotiation.cc:407] Connection header received
0927 07:12:16.578301 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.578306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.578855 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.579010 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.579546 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.580413 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.582580 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.583200 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.583344 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.584207 (+ 863us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.584854 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.584974 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.585072 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.585153 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15022,"thread_start_us":15595,"threads_started":1}
I20260927 07:12:16.600018 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.591903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.30.16.125:40349)
0927 07:12:16.592330 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.592340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.593446 (+ 1106us) server_negotiation.cc:407] Connection header received
0927 07:12:16.593708 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.593713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.593808 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.593878 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.594801 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.595893 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.597728 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.598370 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.598515 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.599173 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.599317 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.599406 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.599581 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.599669 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 07:12:16.601331 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.591892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.30.16.126:39637)
0927 07:12:16.592929 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.592935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.593428 (+ 493us) server_negotiation.cc:407] Connection header received
0927 07:12:16.593728 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.593734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.594176 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.594376 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.595241 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.596196 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.598476 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.598989 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.599139 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.599729 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.600262 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.600359 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.600922 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.601008 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":604,"threads_started":1}
I20260927 07:12:16.601899 21532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 07:12:16.593707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.30.16.124:41289)
0927 07:12:16.593908 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 07:12:16.593914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 07:12:16.595391 (+ 1477us) server_negotiation.cc:407] Connection header received
0927 07:12:16.595553 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 07:12:16.595559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 07:12:16.595660 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 07:12:16.595758 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 07:12:16.596470 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.597767 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.599917 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 07:12:16.600520 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 07:12:16.600663 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 07:12:16.601100 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 07:12:16.601268 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 07:12:16.601378 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 07:12:16.601540 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 07:12:16.601625 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 07:12:16.606150 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 21127
I20260927 07:12:16.607323 21189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:16.720403 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 21127
W20260927 07:12:16.742002 21293 connection.cc:535] server connection from 127.0.0.1:44100 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 07:12:16.742404 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 21201
I20260927 07:12:16.743611 21263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:16.854427 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 21201
I20260927 07:12:16.872028 30785 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu pid 21276
I20260927 07:12:16.873257 21338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 07:12:16.983978 30785 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhkCnwT/build/asan/bin/kudu with pid 21276
2026-09-27T07:12:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5133 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9622 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (368993 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST